CMX G3.5 — New Layer in the Memory Hierarchy
Memory Architecture

CMX G3.5 — New Layer in the Memory Hierarchy

NVIDIA's Context Memory eXtension inserts between DRAM and NVMe for KV cache management
Memory Hierarchy — Speed vs Capacity
T0
GPU HBM Registers
On-chip · Rubin 22 TB/s
<1 ns
fastest
T1
GPU HBM4 Memory
288 GB · 22 TB/s
~10 ns
T2
CPU LPDDR5X DRAM
SOCAMM2 · 1.2 TB/s
~70 ns
NEW
G3.5
CMX Flash Memory
Ethernet-attached NVMe
μs range
T3
Local NVMe SSD
Standard server storage
~100 μs
T4
Enterprise Storage
Object / block / tape
ms range
slowest
KV Cache Lifecycle — Four-Step Flow
1
Inference Completes
Rubin GPU finishes generating tokens. KV cache fills HBM. Next request incoming.
2
BlueField-4 Offloads to CMX
DPU automatically moves completed KV cache from HBM to G3.5 CMX storage, freeing GPU memory for next task.
3
Grove Routes Next Request
Load balancer identifies which physical node holds the user's cached context, routes the new request there.
4
NIXL Pre-stages Cache to HBM
NVIDIA Inference Transfer Library prefetches KV cache from CMX back to GPU HBM before inference begins — zero cold-start penalty.
5×
Real-time token generation throughput improvement from CMX + NIXL + Grove vs. conventional infrastructure.
Long-context agentic sessions no longer penalized by KV cache eviction.