The Unspoken War Behind GPUs
NVIDIA has entered the CPU market.
And yet, no one seems to find it strange.
In March 2025, Jensen Huang took the GTC stage and announced Vera CPU, stating:
“We are now building AI factories.”
The audience applauded. But buried within that announcement was a true declaration.
The world’s largest semiconductor company had walked headlong into the data center CPU market, a territory dominated by Intel and AMD for decades. Intel’s stock fell 3.2% the next day. AMD dropped 1.8%. And that was that. News cycles were filled with other things days later.
However, there’s a quiet number.
In its first year, Vera CPU’s pre-orders exceeded $20 billion.
Considering Intel’s 2024 annual revenue for its data center segment was $12.8 billion, this wasn’t just a new product launch. Something was changing. Fundamentally.
1. The Bottleneck We Didn’t See
For the past five years, the narrative of the AI infrastructure war has been simple.
GPUs are important.
More VRAM, higher FLOPS, faster interconnects. H100, A100, Blackwell. Trillions of dollars in investment flowed in that direction alone.
But starting in 2024, something strange began to be observed.
As data center operators started measuring the actual utilization of their GPU clusters, an uncomfortable number emerged.
In agentic workflows—AI systems that plan multiple steps themselves and invoke tools—the actual computational utilization of GPUs was less than half of expectations.
The GPUs were idle. Waiting.
Waiting for what? -> Waiting for the CPU’s help.
This wasn’t just a bottleneck. It was a crack in the architecture.
-> And NVIDIA had been seeing this crack for three years.
Consider how agentic AI works.
When a user inputs, “Analyze this contract, search for relevant case law, and draft a summary report,”
The AI doesn’t just generate text.
It forms a plan. -> Selects tools. -» Invokes APIs. -> Interprets results. -0> Decides the next step.
All this orchestration happens on the CPU, not the GPU.
The problem is that the proportion of this orchestration is growing.
In agentic workflows with an extremely high proportion of tool invocation, analyses have shown that 50% to as much as 88% of the total latency arises from CPU tool processing delays.
The remaining 12% to 50% is the actual inference processed by the GPU.
Trillion-dollar GPU clusters were spending more than half their time idle, waiting for general-purpose CPUs to finish processing.
This is what NVIDIA was seeing. And NVIDIA decided to fill that bottleneck themselves.
2. Agentic AI Has Changed the Grammar of Computation
The distance between a chatbot and an agent is much greater than you might think.
A chatbot takes input and produces output through a one-way transformation.
The essence of this computation is massive matrix multiplication, which GPUs excel at.
They process tokens in parallel, multiply weights, and predict the next token.
In this process, the CPU’s role is minimized to that of a manager, relaying instructions and receiving results.
Agents are different.
While an agent handles a single user request, it generates at least 15 times more tokens compared to a chatbot.
The reason is simple.
With every decision, the agent writes an internal monologue.
“This is the current situation. The next thing to do is that. I will invoke this tool like this.”
This plan-execute-verify loop repeats, causing context to explode.
More importantly, that 15x is not uniform.
Many of the computations an agent performs are sequential, with complex branching and unpredictability.
Which API to call, how to interpret its results, and which path to take for recovery if an error occurs—all this control flow is the domain of the CPU, which is strong in complex branch prediction and sequential execution, not the GPU, which is optimized for parallel processing.
This is a shift in the computational landscape.
AI computation is moving from ‘massive parallel matrix operations’ to ‘a repetitive combination of complex sequential control + massive parallel operations’.
This has implications for data center design.
Existing AI training and inference data centers were structured to operate with approximately 30 million general-purpose CPU cores per gigawatt.
To handle the orchestration load of agentic inference, this density needs to be increased to approximately 120 million cores per gigawatt—about a fourfold increase, according to analyses.
It’s not just about needing more CPUs.
The nature of what CPUs do has changed.
From general-purpose computation units to high-density control infrastructure specialized for AI orchestration.
And CPUs designed for that role simply did not exist before.
3. Olympus: A Design That Rejected Generality
Remember NVIDIA’s previous CPU, the Grace CPU?
Released in 2023. Based on Arm Neoverse V2 cores. It came out quietly and was used quietly.
Grace was a safe approach, using an existing core and licensing Arm’s off-the-shelf design.
Vera is different.
The core of Vera is the ‘Olympus’ custom core.
It’s the first CPU core designed entirely by NVIDIA. It’s compatible with the Armv9.2 instruction set, but its internal structure has been completely redesigned.
Why rebuild from scratch?
The answer to this question explains Vera’s entire design philosophy.
General-purpose CPU cores are designed to handle a variety of workloads. Word processors, databases, games, compilers. For this generality, branch predictors are optimized for typical code patterns. Cache structures are designed to handle diverse memory access patterns.
The code executed by agentic AI deviates extremely from these ’typical’ patterns. Python virtual machines, virtualized sandboxing tools, distributed database query schedulers—these create irregular and heavily twisted control flows.
General-purpose branch predictors miss more often, and pipeline stalls become frequent.
The branch predictor and instruction decoder pipeline of the Olympus core have been customized for this pattern.
To interpret and process irregular sequential code flows with the shortest latency. The result: a 1.5x improvement in instructions per clock (IPC) compared to the previous Grace core.
However, Olympus’s more interesting innovation is ‘Spatial Multithreading.’
If you’re familiar with Intel’s Hyper-Threading, you can easily understand what this is. And you can understand why it’s insufficient.
Hyper-Threading allows two logical threads to share a single physical core in a time-sharing manner, where one thread executes while another waits for memory.
. However, in situations where two threads require computation simultaneously—like an agent invoking two tools at the same time and processing their results—
Competition arises for the compute units. Jitter occurs. Response times become irregular.
Olympus’s spatial multithreading takes a different approach.
It permanently partitions the 88 physical core’s compute resources into two completely independent sections at the hardware level.
The 176 virtual threads cannot encroach on each other’s resources.
Consistent performance at the individual physical thread level is guaranteed.
Even if thousands of independent agents simultaneously invoke compilers and run sandboxes,
The execution delay of a specific agent does not prevent the processing of other agents—or even the entire GPU cluster.
4. Monolithic, Not Chiplet: The Radical Reason for a Conservative Choice
Vera CPU adopts a single monolithic die design. This is a choice that goes against current trends.
AMD’s EPYC processors use a chiplet architecture. A large die is split into smaller pieces and connected via interconnects on a package substrate. The advantages of this approach are clear. Yield increases. Costs decrease. Core counts can be increased without increasing die size. This was the weapon with which AMD chased and overtook Intel.
However, the chiplet architecture has a hidden cost. Every time data crosses between different silicon pieces, a tiny delay occurs. A delay measured in nanoseconds, but which accumulates into a significant bottleneck in agentic AI orchestration.
This problem doesn’t exist with monolithic dies. All cores are on the same piece of silicon. The variation in transmission latency due to physical distance is perfectly controlled.
The 88 Olympus cores are connected by the ‘2nd Gen Scalable Coherency Fabric (SCF)’. It provides a bisection bandwidth of 3.4 TB/s across the entire die for communication. Cores can access caches or I/O ports on the opposite side of the die at consistent speeds.
And memory. Here, Vera makes another counter-intuitive choice.
Conventional server CPUs use DDR5. It’s standard, proven, and the infrastructure is in place. However, DDR5’s bandwidth limitations and power consumption become a bottleneck in agentic workloads. Top-tier x86 server CPUs consume over 100W just to drive the memory bus to achieve bandwidth of around 600 GB/s with 12+ channels of DDR5.
Vera has chosen LPDDR5X. Originally memory for mobile devices. Its advantage is low power consumption. It has been adapted for servers. Mounted directly onto server motherboards in a new form factor called SOCAMM2.
Result: 1.2 TB/s memory bandwidth. Twice that of DDR5. Power consumption under 30W. One-third that of DDR5.
This saved 70W is converted into compute core performance.
[IMAGE: Side-by-side physical comparison of SOCAMM2 LPDDR5X module and standard DDR5 DIMM on a neutral matte surface — showing the significant form factor size difference, screw-compression mounting mechanism visible on the SOCAMM2, and server motherboard memory slot placement context. Clean product photography with technical precision. Style: technical documentation photography, neutral gray background, side-lighting to emphasize component geometry.]
5. What the Numbers Say
Benchmarks can be contentious. However, data empirically measured by Phoronix at NVIDIA’s headquarters has sent significant shockwaves through the computer hardware industry.
Single-socket configuration, 450W TDP setting. Linux kernel source code compilation test.
Vera CPU (88 cores): Completed in 20 seconds.
Time taken by 128-core server CPUs for the same test: 40 seconds.
This means it’s twice as fast with less than half the cores, implying four times the per-core computational efficiency.
Looking more broadly: In geometric mean performance, Vera outperformed AMD EPYC 9575F (64 cores, Zen 5) by 10% and Intel Xeon 6980P (128 cores) by 55%.
When these numbers are translated into power efficiency, the picture becomes clearer. Intel and AMD achieve this performance while consuming over 500W. Vera uses 450W plus 30W for memory. This results in about a 2x advantage in performance per watt.
And this directly connects to data center operating costs. Power is a core operating expense. Double the performance for the same power means obtaining the same performance in half the space, with half the cooling costs.
This is why hyperscalers pre-ordered $20 billion.
[INFOGRAPHIC: “CPU Performance Benchmark — NVIDIA Vera vs x86” — Horizontal bar charts showing Linux kernel compile time (Vera: 20s vs 128-core baseline: 40s), overall benchmark geometric mean (Vera 100% / AMD EPYC 90.9% / Intel Xeon 64.5%), and a three-column power efficiency breakdown comparing TDP and memory bus wattage across all three chips. Placed here to anchor the performance claims made in the preceding paragraphs with quantitative evidence.]
6. But One CPU Isn’t Enough
Even though the performance of the Vera CPU is impressive, it alone cannot solve the fundamental bottleneck of agentic AI.
There’s a second bottleneck: context memory.
As agents perform complex tasks, context accumulates. Previous conversations, execution history, intermediate results—these are collectively called ‘KV cache (Key-Value Cache)’. If an agent continues its work over several days, this KV cache can swell to tens or hundreds of gigabytes.
Where is this stored?
Storing it in the GPU’s internal HBM memory is the fastest. However, HBM is expensive and has limited capacity. If that space is filled with KV cache, there’s less memory left for actual inference. It’s wasteful.
On the other hand, putting it on a general SSD takes time to retrieve later. The fast conversational flow breaks abruptly.
This is the ‘context memory storage barrier.’ And NVIDIA has designed a new memory tier to address this.
It’s called G3.5.
Memory tiers are usually thought of like this: CPU Registers → L1/L2/L3 Cache → DRAM → SSD → HDD. The faster it is, the more expensive and smaller the capacity. The slower it is, the cheaper and larger.
G3.5 fits somewhere in between these tiers. A dedicated flash memory area connected via Ethernet, slower than DRAM but much faster than a general SSD. This is CMX (Context Memory eXtension).
[INFOGRAPHIC: “CMX G3.5 — New Layer in the Memory Hierarchy” — Vertical tier stack from T0 (GPU registers, >100 TB/s) down to T4 (enterprise storage, ms latency), with G3.5/CMX highlighted in orange as the newly inserted layer between DRAM and local NVMe. Includes a four-step KV cache lifecycle flow: inference completes → BlueField-4 offloads to CMX → Grove routes next request → NIXL pre-stages cache to HBM. Placed here immediately after the explanation of the context memory bottleneck to visualize the solution architecture.]
As the agent finishes its work and the GPU processes the next request, the previous KV cache is automatically moved from HBM to CMX. If the same user reconnects, the cache is proactively loaded from CMX to GPU memory—just before actual inference begins.
This proactive loading is handled by NIXL (NVIDIA Inference Transfer Library). And Grove, the load balancer, determines which node to send the request to—the node physically closest to the user’s previous context.
The combination of these three elements—CMX + NIXL + Grove—has been measured to improve real-time token generation throughput by up to 5 times compared to existing infrastructure.
5x is an impressive number. But more important is the direction. NVIDIA isn’t just making faster chips; they are replacing the entire software and hardware stack of AI infrastructure with their own designs.
7. Vera Rubin NVL72: The Data Center as a Single Chip
While the Vera CPU is sold individually, its true purpose lies elsewhere.
‘Vera Rubin NVL72’. This name has become one of the most frequently mentioned terms in the AI infrastructure industry in 2025.
The NVL72 is a single, integrated rack. Inside this rack are:
72 Rubin GPUs. 36 Vera CPUs. BlueField-4 DPUs. ConnectX-9 SuperNICs. NVLink 6 switches. Spectrum-6 Ethernet switches. And in December 2025, Groq’s LPUs, acquired for $20 billion.
A total of seven silicon components. NVIDIA calls this ‘Extreme Co-design.’
This means designing the entire rack as a single computer from the ground up, rather than designing and combining individual chips. The NVLink-C2C interconnect connects CPUs and GPUs at 1.8 TB/s, seven times faster than PCIe Gen 6. At this bandwidth, data transfer latency between CPU and GPU eliminates GPU idle time.
Looking at the Rubin GPU: TSMC 3nm process. 336 billion transistors. 288GB HBM4 memory, 22 TB/s bandwidth. Single-chip inference performance of 50 PetaFLOPS (PFLOPS)—5 times that of Blackwell.
And the Groq LPU.
The Groq is a chip designed with a unique philosophy. To eliminate bus bottlenecks caused by frequent communication between GPUs and large HBM memory, most of the silicon die is filled with high-speed SRAM. Approximately 500MB of on-chip SRAM, 80 TB/s memory bandwidth. And deterministic execution—an architecture where the compiler perfectly plans the timing of all operations beforehand, making complex hardware-level branch prediction unnecessary.
What is this characteristic strong at? The decode stage, which generates tokens sequentially and rapidly.
To understand the inference flow of the NVL72 rack, LLM inference must be divided into two stages.
Prefill: The stage where the input prompt is processed all at once. This requires large-scale parallel matrix operations. Rubin GPUs are responsible for this.
Decode: The stage where, after the first token is generated, subsequent words are generated sequentially. This is sequential and memory bandwidth sensitive. Groq LPUs are responsible for this.
When Rubin finishes the prefill, the computational context is handed over to the Groq LPU. The Groq’s 80 TB/s SRAM bandwidth generates tokens at the speed of light.
The result of this separation: The Vera Rubin NVL72 achieves a 35x improvement in inference output per watt for trillion-parameter models compared to Blackwell.
35x. Everyone knows this is not just an incremental generational improvement.
[INFOGRAPHIC: “Vera Rubin NVL72 — 7-Silicon Extreme Co-Design Stack” — Vertical flowchart showing data flow across four chip tiers: Vera CPU (control plane) → Rubin GPU (prefill) → Groq 3 LPU (decode) → BlueField-4 DPU (storage control). Interconnect bandwidth labels on each arrow. Three-step inference pipeline summary and 35× performance-per-watt highlight. Placed here to consolidate the NVL72 architecture into a single visual reference after all components have been introduced narratively.]
[IMAGE: Wide-angle photograph of a fully populated NVL72-style server rack in a data center environment — dense GPU and CPU modules visible behind perforated front panel, status LEDs creating a grid of blue and amber light points, thick power and fiber cables entering from top and bottom. The rack occupies the full frame height, suggesting contained yet enormous computational density. Style: technical documentary photography, available light with LED-colored ambient, shallow depth of field focused on middle rack section.]
8. A $200 Billion Territory Declaration
Jensen Huang made a calculation at GTC 2025. There are currently about 1 billion general-purpose PCs in the world. Each PC has a CPU. Just as humans have CPUs in their work devices, infrastructure where AI agents work also needs CPUs.
Tens of billions of AI agents. CPUs to provide sandboxed computation for each.
If the structure of the market Intel and AMD have monopolized for decades changes, the size of that vacant space is $200 billion. Jensen Huang’s declaration.
Is it an exaggeration? Perhaps. But the fact remains that $20 billion in pre-orders came in the first year.
Intel’s 2024 annual revenue for its data center and AI segment was $12.8 billion. It already surpassed that in its first year. Even if this is a market that started with exaggeration, the direction is correct.
However, a more important question arises here. Why is NVIDIA making CPUs a threat to Intel and AMD? NVIDIA is a GPU company. Isn’t making a new CPU just adding another competitor?
There’s a reason why that perception is wrong.
NVIDIA’s Vera CPU can be sold independently, but it offers a completely different value proposition when integrated into the NVL72. Using that CPU offers 7x faster connection to GPUs, optimized integration with CMX, and seamless operation of the DOCA Memos software stack. Outside the NVIDIA ecosystem, these optimizations cannot be obtained.
This is the essence of vertical integration. The CPU itself is not the goal, but a wedge that blocks escape routes.
Hyperscalers have had choices until now. While they might have been forced to buy GPUs from NVIDIA, they have used AMD or their own designs for CPUs. This was the last bastion of infrastructure control.
The NVL72 targets that bastion.
9. Resistance From Three Directions
Naturally, this offensive is met with resistance. From three directions simultaneously.
The Arm-Meta Alliance
In 2025, Arm CEO Rene Haas announced the ‘AGI CPU’ reference architecture jointly with Meta’s infrastructure team. It packs 136 of their flagship ‘Neoverse V3’ cores into a single package. Under 300W. Capable of operating 8,160 physical cores per rack in a standard air-cooled setup.
Key message: Higher core density can be achieved with an open architecture, without NVIDIA’s high-temperature liquid cooling infrastructure.
Reduced capital expenditure per gigawatt: Up to $10 billion.
This is a logic of economic independence rather than technical superiority. It signifies that the core density demands of agentic orchestration can be met without NVIDIA’s vertical lock-in.
Acceleration of Cloud Big 3’s Internalization
Google’s Axion. Amazon’s Graviton 5. Microsoft’s custom-designed chips. All of them are accelerating the trend of processing their services’ orchestration loads with their own silicon instead of external commercial CPUs.
Amazon’s Graviton and Nitro hybrid systems have already achieved an annual revenue run rate of $20 billion. Google’s Axion claims a 65% improvement in performance-per-dollar and a 60% improvement in energy efficiency compared to existing solutions.
Their strategy is simple. Using the scale of their cloud services as leverage, they aim to reduce costs that would otherwise be paid to NVIDIA and pass those benefits on to cloud users through price reductions.
Geopolitical and Regulatory Headwinds
Jensen Huang acknowledged at Computex that NVIDIA’s $200 billion TAM forecast includes Chinese demand. However, the U.S. Department of Commerce’s export controls are blocking direct channels for high-spec chipsets into China. China is pushing to foster its domestic AI ecosystem and strengthening its stance on deferring U.S. semiconductor imports.
Furthermore, the $20 billion Groq acquisition structure became controversial. Allegations that the de facto merger through a ’licensing agreement and personnel hiring’ intentionally circumvented FTC merger review have led senators like Edward Markey and Elizabeth Warren to officially request an investigation.
It remains an open question how much these three forms of resistance will slow down NVIDIA’s expansion.
[IMAGE: Split-composition editorial photograph — left half shows a dense circuit board with NVIDIA branding visible on a chip package; right half shows an Arm logo on a pale gray surface, deliberately out of focus and receding. The composition implies competition and divergence of paths. Style: editorial business photography, neutral background, high contrast, symbolic rather than literal framing.]
10. What Happens When Monopoly is Complete
There’s a pattern in computing history. When vertical integration creates extreme performance, many choose dependence in the face of that performance. And when that dependence becomes deep enough, the counter-attack of the open ecosystem begins.
It happened with IBM’s mainframes. It happened with DEC’s VAX. It happened with Intel-Microsoft’s Wintel. It happened with Apple’s vertical integration.
In each case, the vertical integration empire dominated for a long time. But in each case, the open ecosystem eventually found a way—a space that the empire didn’t control, or a space that the empire made too expensive.
The scale of what NVIDIA is building now is different from before. Vertically integrating CPUs, GPUs, DPUs, network switches, memory storage, and software stacks simultaneously is not a monopoly in a single product area, but a monopoly over the entire layer of computing infrastructure.
Reports have emerged that even within OpenAI and Anthropic, internal discussions are underway concerning concerns about locking global inference compute power into a specific hardware vendor’s ecosystem.
Whether this is a warning or a misplaced concern, time will tell. But the very existence of this concern is a significant signal. It indicates that the cost of dependence is being calculated before surrendering to performance.
NVIDIA is not a company selling GPUs anymore. It is trying to become a company that designs the entire computing infrastructure for the agentic AI era. The Vera CPU is the most explicit declaration of that ambition.
11. The Grammar of Infrastructure is Being Rewritten
Let’s step back for a moment.
What are we witnessing?
NVIDIA’s Vera CPU is an excellent chip. But that alone doesn’t explain why this story is important. Faster CPUs always emerge.
This is different because this chip is evidence that the design principles of AI infrastructure itself are changing.
The old design principle was this: Add AI accelerators on top of general-purpose CPUs. Connect them with general-purpose interconnects. Store data in general-purpose storage. Each layer was independent and replaceable.
The new design principle is different: Design all layers together to suit the characteristics of AI workloads. Prioritize optimization over replaceability. And a single vendor monopolizes the authority of that optimization.
This transition is also a philosophical shift in computing infrastructure. The long-standing tension between the principles of open systems—generality and replaceability—and the principles of vertical integration—optimization and efficiency.
That tension has reached a new phase in the age of AI. The extreme performance demands of AI have created a realm unattainable by general-purpose systems. And it’s an era where those who reach that realm first can dominate the market.
The Vera CPU is a move towards that realm. And the fact that this move has begun, regardless of whether it succeeds or fails, is rewriting the future of computing infrastructure.
12. The World Where Agents Work
Finally, from a slightly different angle.
Imagine the “tens of billions of AI agents” Jensen Huang spoke of. Each working in its own sandbox, with its own memory, using its own tools. Like human workers, they are assigned specific tasks, report results, and await the next task.
In this world, what is a CPU? It’s what provides thinking space for the agent. It’s what provides hands to use tools. It’s what provides a path to store and retrieve memories.
Just as human workers need desks and computers, AI agents need CPUs.
And who designs that CPU determines what agents can do—and what they cannot.
NVIDIA is not making the Vera CPU to fill a gap in the market. It’s a declaration to design the computing infrastructure of the agentic AI era from the ground up.
Where do Intel and AMD stand before that declaration? What are Arm, Google, and Amazon preparing? And how will the outcome of all these battles determine the capabilities and costs of AI agents?
That is the reality of the war quietly unfolding.
Behind the GPUs.
[IMAGE: A single empty office desk at dusk, seen through floor-to-ceiling glass — computer monitor glow visible on the desk surface, city skyline blurred in the background. The desk is unoccupied but the screen is active. The image implies presence without a human body: an agent at work. Contemplative and slightly uncanny. Style: architectural interior photography, available light from screen and city, muted palette, wide-angle lens with slight distortion at frame edges.]
References
- NVIDIA Corporation, “Vera Rubin Platform Architecture Whitepaper” (Jensen Huang, 2025)
- Michael Larabel, “NVIDIA Vera CPU Benchmarks on Linux,” Phoronix (2025)
- Jonathan Ross, “A Software-Defined Tensor Streaming Processor for Large-Scale Machine Learning,” Groq Inc.
- Arm Editorial Team, “Arm Everywhere: Accelerating Agentic AI in Modern Datacenters” (Rene Haas, 2025)
- Vincent Hsu, “IBM Storage Scale integration with NVIDIA Dynamo Serving Engines,” IBM Systems Group (2025)
- NVIDIA Developer Documentation, “DOCA Memos API Reference Guide” (2025)
- NVIDIA Research, “Agentic AI Orchestration Bottleneck Analysis: CPU vs GPU Latency Distribution” (2024)
- Tom’s Hardware, “NVIDIA Vera CPU vs Intel Xeon 6980P vs AMD EPYC 9575F: Benchmark Comparison” (2025)
- IEEE Computer Architecture Letters, “Spatial Multithreading for Agentic Workloads: Design Principles and Performance Analysis” (2025)
- Semiconductor Engineering, “SOCAMM2 Form Factor: Enabling LPDDR5X in Server Environments” (2025)
- Next Platform, “Understanding NVIDIA’s CMX Context Memory Extension Architecture” (2025)
- Hot Chips 37, “NVIDIA Olympus Microarchitecture Deep Dive” (2025)
- Solidigm Technical Brief, “D7-PS1010 NVMe SSD for AI Inference KV Cache Workloads” (2025)
- Reuters, “U.S. Senators Request FTC Investigation into NVIDIA-Groq Acquisition Structure” (2025)
- Dylan Patel, “The Agentic AI Infrastructure Wars: CPU, Memory, and the Battle for the Control Plane,” SemiAnalysis (2025)