NVIDIA says full-stack codesign raises AI inference output per HBM byte

NVIDIA said on Nov. 5 that designing GPUs, CPUs, networking, storage and software together, rather than adding more memory, raised agentic inference throughput per dollar by up to 30x between GPU generations.

The company's newsroom post1 says larger models, longer contexts and agentic applications are increasing GPU memory pressure while supply tightens and prices rise. All performance figures in it are NVIDIA's own, and the post gives no configuration or cost basis for them.

NVIDIA frames the problem this way. Frontier models now have trillions of parameters, and agentic trajectories carry prior context across repeated model and tool calls, in some cases processing up to 3 million tokens. During inference, GPU high-bandwidth memory (HBM) mainly holds model parameters and the KV cache, which stores the context for each active request.

The post argues that adding HBM capacity alone eventually stops helping because bandwidth limits how fast data reaches the compute units. It calls this limit the "memory wall" and says capacity beyond that point still adds cost and power draw. NVIDIA's answer is a memory hierarchy that moves data off HBM when it is not in active use.

The headline figures come from the SemiAnalysis AgentX DeepSeek V4 Pro workload. NVIDIA said GB300 NVL72 delivers up to 10x higher throughput per dollar than the H200 platform1 at 100 tokens per second per user, with HBM capacity per GPU roughly doubling. At 125 tokens per second per user, it said Vera Rubin NVL72 delivers up to 30x higher throughput per dollar than GB300 NVL72, with HBM capacity per GPU up about 3%.

The post does not state the precision, software versions or cost basis behind either "up to" figure. It also does not say whether the 10x comparison between 72-GPU racks and H200 systems is like-for-like, or whether SemiAnalysis published or validated the AgentX results.

NVIDIA's technical blog3 on the same theme reports different numbers. It says Vera Rubin NVL72 delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user. That is a different metric and operating point from the newsroom post's per-dollar figure at 125 tokens per second per user, so the two 30x claims are not the same result.

The documents also differ on capacity. The newsroom post says HBM per GPU rose roughly 3% from Blackwell Ultra to Rubin, while the technical blog says capacity "held at 288 GB" from GB300 to Rubin.

The newsroom post describes techniques for three kinds of data:

• Model weights. NVIDIA says its NVFP4 format reduces the space weights need compared with FP8. The technical blog puts the reduction at approximately 1.8x versus FP8 and 3.5x versus FP163. A separate NVIDIA blog on NVFP42 says the format met MLPerf closed-division accuracy requirements on DeepSeek-R1, Llama 3.1 8B and 405B, and Llama 2 70B. The newsroom post lists models released with NVFP4 checkpoints1, including FLUX.2 [klein], Inkling and NVIDIA Nemotron 3.5 Super VL. It describes them as "demonstrating growing model-builder adoption."
• KV cache. The newsroom post says NVIDIA Dynamo software moves inactive context to NVIDIA CMX context memory storage, a shared tier it says costs far less per bit than HBM, and returns it before reuse. The technical blog says each Vera CPU provides up to 1.5 TB of LPDDR5X, connected coherently to Rubin at 1.8 TB/s over NVLink-C2C, to hold idle cache during tool calls.
• Retrieval data. According to the newsroom post, Nemotron Retriever models produce smaller embeddings, and the cuVS library lets the index be built and served across CPU and storage so the knowledge base never occupies HBM.

The newsroom post says that within an NVL72 system, weights can be distributed across up to 72 GPUs. The technical blog separately describes wide expert parallelism in TensorRT-LLM, which spreads mixture-of-experts weights across eight or more GPUs to leave more HBM per GPU for cache.

The technical blog lists Rubin bandwidth of 19.2 TB/s. Dataconomy reported on Nov. 24 that NVIDIA lowered the published per-GPU HBM4 bandwidth for Vera Rubin to 19.2 TB/s from 22 TB/s, and NVLink 6 bandwidth to 3 TB/s from 3.6 TB/s. It said NVIDIA did so without announcing or explaining the changes. Dataconomy also noted that NVIDIA's per-GPU table still lists 22 TB/s. It cited an NVIDIA spokesperson as saying the company's HBM4 memory partners remained on track for production shipments. The report is from a single outlet, and the newsroom post does not mention the revision.