Switchboard Logic Behind Agentic Inference Economics

I notice the break immediately: plain chat ticks like a metronome, while agent jobs sprawl like a busy kitchen . A model sketches a plan, invokes utilities, studies outcomes, then loops again with revised context. One-shot answers belong to an earlier phase; current systems haul growing state forward, so token volume climbs fast.

The figure that sticks with me equals roughly fifteenfold consumption. OpenRouter examined 100T tokens of live traffic and found solo agent requests burning around 15x tokens versus casual chat. I picture flat area turning into solid volume . A lead coordinator also delegates slices to subordinate helpers. During one authentic software session, retained context expanded from 60K toward 400K because each round carried history onward.

Fixed yardsticks miss living jobs

Old yardsticks feel narrow to me. Sending 8K input plus 1K output and quoting speed explains isolated turns, not living assignments. Fresh measurement replays authentic software sessions, covering nearly 99K model calls. Input size shifts, output size shifts, pauses around utility gaps stay intact. Waiting contexts remain resident, so cached state stays warm. Every platform receives identical records, which isolates hardware plus serving software as the differentiator.

I frame the problem around factory output: useful agent accomplishment per fixed power budget. Raw efficiency alone can mask sluggish interaction, so I pair it with visible speed, end-to-end delay, and initial responsiveness. Adding concurrency usually slows each participant, while sparse loading wastes capacity. Champion setups maximize productive throughput per megawatt without sacrificing liveliness. Agent tempo, model behavior, and token flow deserve joint reading.

Full-stack coordination creates headroom

Gains come from orchestration rather than isolated silicon tweaks; I hear an orchestra , not a soloist. NVLink binds 72 accelerators into one rapid domain, letting giant expert models act as a unified system . Dynamo splits early processing from token generation, scales each pool separately, and steers fresh rounds toward workers holding relevant cached state. TensorRT-LLM adds NVFP4 precision plus broad expert parallelism, keeping execution units occupied while thousands of agents proceed concurrently. Nvidia engineering notes describe this blueprint, and SemiAnalysis replay numbers reflect it.

Placed side by side, published figures tell a clear story. GB300 NVL72 delivers up to 15x factory throughput per megawatt over H200 NVL8 on DeepSeek V4 Pro work, with cost per million tokens down up to 10x. Vera Rubin NVL72 reaches up to 30x over GB300 NVL72 near 160 tokens per second per user. Larger setups widen the spread: on Kimi K3 2.8T, GB300 NVL72 shows roughly 80x over H200 NVL8 and stretches interaction near 215 tokens per second per user, beyond prior reach. CloudNews summaries echo these Nvidia measurements pending SemiAnalysis review.

My lesson stays simple: agents form the baseline workload now, not an edge case. Serving them well demands system-level design. I would separate context-heavy paths, favor cache-aware routing, and track power efficiency alongside interaction quality. Newcomers can start from Nvidia use-case guides plus SemiAnalysis pages, with CloudNews recaps for compact numbers. When agent rhythm, model conduct, and token movement get weighed together, priorities settle quickly.