NVIDIA has entered the CPU market.
Yet no one finds it strange.
In March 2025, Jensen Huang took the stage at GTC to announce the Vera CPU, stating:
“We are now building AI factories.”
The audience applauded. But buried beneath that announcement lay the true declaration.
The world’s largest semiconductor company walked head-on into the data center CPU market—a domain monopolized by Intel and AMD for decades. Intel’s stock fell 3.2% the next day. AMD dropped 1.8%. That was it. Days later, the news cycle moved on to other topics.
Yet there is one quiet figure.
In its first year of release, pre-orders for the Vera CPU exceeded $20 billion.
Considering Intel’s total data center revenue in 2024 was $12.8 billion, this was no ordinary product launch. Something was shifting. Fundamentally.
1. The Bottleneck We Failed to See
For the past five years, the narrative of the AI infrastructure war was simple.
GPUs are what matter.
More VRAM, higher FLOPS, faster interconnects. H100, A100, Blackwell. Tens of billions of dollars flowed entirely in that direction.
However, starting in 2024, something strange began to be observed.
When data center operators began measuring the actual utilization rates of their GPU clusters, uncomfortable figures emerged.
In agentic workflows—AI systems that autonomously plan multiple steps, invoke tools, and execute code—the actual computational utilization of GPUs did not even reach half of what was expected.
The GPUs were idling. Waiting.
What were they waiting for? -> They were waiting for the CPU.
This was not a simple bottleneck. It was an architectural fracture.
-> And NVIDIA had seen this fracture coming three years prior.
Consider how agentic AI operates.
When a user prompts, “Analyze this contract, search for relevant case law, and draft a summary report,”
the AI does not merely generate text.
It makes a plan. -> Selects tools. -» Calls APIs. -> Interprets results. -> Decides the next step.
All of this orchestration takes place on the CPU, not the GPU.
The problem is that the weight of this orchestration is growing rapidly.
In agent workflows with extremely heavy tool-calling, analysis revealed that 50% to as much as 88% of total latency originates from CPU tool-processing delays.
The remaining 12% to 50% is the actual inference handled by the GPU.
Multi-billion-dollar GPU clusters were spending more than half their time idling, waiting for general-purpose CPUs to finish processing.
This is what NVIDIA was seeing. And NVIDIA decided to solve that bottleneck themselves.
2. How Agentic AI Changed the Grammar of Computing
The gulf between chatbots and agents is far wider than thought.
A chatbot takes an input and produces an output via a unidirectional transformation.
The essence of that computation is massive matrix multiplication, which is precisely what GPUs excel at.
It processes tokens in parallel, multiplies weights, and predicts the next token.
In this process, the CPU’s role is minimized to that of an administrator passing instructions and receiving results.
Agents are different.
While an agent processes a single user request, it generates at least 15 times more tokens than a chatbot.
The reason is simple.
Agents write an internal monologue with every decision.
“The current status is this. What needs to be done next is that. I will call this tool like this.”
As this plan-execute-verify loop repeats, context expands exponentially.
More importantly, that 15x increase is not uniform.
Much of the computation handled by an agent is sequential, heavily branched, and unpredictable.
Which API to call, how to parse the response, which recovery path to take upon error—all of this control flow belongs not to GPUs optimized for parallel processing, but to CPUs built for complex branch prediction and sequential execution.
This represents a shift in the computational landscape.
The core of AI computing is shifting from ‘massive parallel matrix arithmetic’ to an ‘iterative combination of complex sequential control and massive parallel compute.’
This introduces major implications for data center architecture.
Legacy AI training and inference data centers operated at a density of around 30 million general-purpose CPU cores per gigawatt.
To handle the orchestration overhead of agentic inference, analyses indicate this density must increase to approximately 120 million cores per gigawatt—roughly a fourfold jump.
It is not simply about needing more CPUs.
The very nature of the work the CPU performs has changed.
From a general-purpose compute processor to high-density control infrastructure specialized for AI orchestration.
And until now, no CPU existed that was designed for that role.
3. Olympus: Architecture Rejecting General Purpose
Do you remember NVIDIA’s previous Grace CPU?
Launched in 2023. Based on Arm Neoverse V2 cores. It arrived quietly and was deployed quietly.
Grace took the safe route of licensing Arm’s off-the-shelf design, adopting proven cores.
Vera is different.
At the core of Vera is the custom ‘Olympus’ core.
As the first CPU core custom-designed in NVIDIA’s history, it is compatible with the Armv9.2 instruction set architecture, but its internal microarchitecture was built entirely from scratch.
Why build it from scratch?
The answer to this question explains the entire design philosophy of Vera.
General-purpose CPU cores are designed to handle diverse workloads: word processors, databases, games, and compilers. For the sake of this versatility, branch predictors are optimized for typical code patterns, and cache hierarchies are designed to accommodate generic memory access patterns.
The code executed by agentic AI deviates radically from these ’typical’ patterns. Python virtual machines, virtualization sandboxing tools, distributed database query schedulers—these produce irregular, deeply contorted control flows.
General-purpose branch predictors mispredict more frequently, leading to constant pipeline stalls.
The branch predictor and instruction decoder pipeline in the Olympus core were customized specifically for these patterns.
They are built to decode and process irregular sequential code flows with minimum latency. The result: a 1.5x improvement in Instructions Per Clock (IPC) compared to the previous Grace core.
Yet an even more fascinating innovation in Olympus is ‘Spatial Multithreading.’
If you are familiar with Intel’s Hyper-Threading, it is easy to understand what this is—and why Hyper-Threading falls short.
Hyper-Threading shares a single physical core between two logical threads via time-slicing, allowing one thread to execute while the other waits for memory.
However, when both threads require active computation at the exact same time—such as an agent concurrently invoking two tools and parsing results—
they compete for execution units. Jitter occurs, and response times become unpredictable.
Olympus’s Spatial Multithreading takes a different approach.
It permanently partitions the compute resources within each of the 88 physical cores into two entirely independent execution domains at the hardware level.
The 176 virtual threads can never encroach on each other’s resources.
They are guaranteed consistent performance on par with dedicated physical threads.
Even when thousands of independent agents concurrently invoke compilers and spin up sandboxes,
execution latency in one agent never stalls another agent—or worse, the entire GPU cluster.
4. Monolithic Instead of Chiplets: Radical Reasons Behind a Conservative Choice
The Vera CPU adopts a single monolithic die design. This choice runs counter to prevailing industry trends.
AMD’s EPYC processors utilize a chiplet architecture. Large dies are broken down into smaller pieces and interconnected across a package substrate. The advantages of this approach are clear.
Yield rates improve.
Manufacturing costs decrease.
Core counts can be scaled up without bloating individual die sizes.
-> This was the very weapon that allowed AMD to catch up to and overtake Intel.
However, chiplet architectures carry a hidden penalty.
Every time data crosses between separate pieces of silicon, micro-latencies are introduced.
Though measured in nanoseconds, when accumulated during agentic AI orchestration, these delays create meaningful bottlenecks.
Monolithic dies eliminate this problem.
All cores reside on the same silicon. Transmission latency variance due to physical distance is perfectly controlled.
The 88 Olympus cores are interconnected via the ‘2nd Gen Scalable Coherency Fabric (SCF).’
Even when a core accesses caches or I/O interfaces on the opposite side of the die, the fabric provides a uniform bisection bandwidth of 3.4 TB/s across the entire die.
Then there is memory. Here, Vera makes another counterintuitive choice.
Conventional server CPUs use DDR5. It is standardized, proven, and backed by established infrastructure.
Yet the bandwidth limits and power draw of DDR5 turn into bottlenecks under agentic workloads.
Top-tier x86 server CPUs consume over 100W just driving the memory bus to achieve around 600 GB/s bandwidth across 12 or more DDR5 channels.
Vera chose LPDDR5X. Originally designed for mobile devices with ultra-low power consumption, it was transplanted into the server realm, mounted directly on server motherboards using the new SOCAMM2 form factor.
The result: 1.2 TB/s of memory bandwidth—twice that of DDR5—at under 30W of power consumption, just one-third of DDR5.
The saved 70W is redirected back into compute core performance.
5. What the Numbers Say
Benchmarks are always open to debate.
However, real-world data measured by Phoronix at NVIDIA headquarters sent significant shockwaves through the computer hardware industry.
In a single-socket configuration set to a 450W TDP, during a Linux kernel source code compilation test:
Vera CPU (88 cores): Completed in 20 seconds.
Time taken by 128-core server CPUs: 40 seconds.
-> With less than half the cores, it delivered twice the speed—meaning four times the compute efficiency per core.
Looking broader: in geometric mean performance, Vera outperformed the AMD EPYC 9575F (64-core, Zen 5) by 10% and beat the Intel Xeon 6980P (128-core) by 55%.
Translating these figures into power efficiency clarifies the picture even further.
Intel and AMD draw over 500W to hit their performance marks.
Vera achieves roughly a 2x advantage in performance per watt, running at 450W with just 30W memory power.
And this directly translates into data center operational expenditure.
Power is the core driver of operating costs.
If you can achieve double the performance at the same power envelope, it means obtaining identical throughput in half the footprint with half the cooling overhead.
This is precisely why hyperscalers committed $20 billion in pre-orders.
6. But a CPU Alone Is Not Enough
Impressive as the Vera CPU’s performance is, it alone cannot solve the fundamental bottleneck of agentic AI.
There is a second bottleneck: Context Memory.
As agents perform complex tasks, context accumulates.
Past conversations, execution history, intermediate results—these are known as the ‘KV Cache (Key-Value Cache).’
When an agent continues working across days, this KV cache balloons to dozens or even hundreds of gigabytes.
Where is this stored?
Keeping it in the GPU’s internal HBM memory is the fastest.
However, HBM is expensive and strictly limited in capacity. Filling that space with KV caches reduces the memory available for actual inference compute—a pure waste.
Conversely, offloading it to standard SSDs causes significant retrieval latency the next time it is accessed, abruptly disrupting rapid conversational flow.
This is the ‘Context Memory Storage Wall.’
To overcome this, NVIDIA engineered a new memory tier designated as G3.5.
The traditional memory hierarchy is typically structured like this:
CPU Registers → L1/L2/L3 Cache → DRAM → SSD → HDD. The faster it is, the more expensive and capacity-limited it becomes.
The slower it is, the cheaper and more capacious.
G3.5 slots right between these tiers.
Slower than DRAM yet substantially faster than standard SSDs, this Ethernet-connected dedicated flash memory domain is known as CMX (Context Memory eXtension).
While an agent concludes a task and the GPU processes subsequent requests, the previous KV cache is automatically migrated from HBM to CMX. When the same user reconnects, it is preemptively fetched from CMX back into GPU memory right before active inference begins.
This preemptive loading is orchestrated by NIXL (NVIDIA Inference Transfer Library).
And routing requests to the optimal node—the physical node closest to the user’s prior context—is determined by the Grove load balancer.
Measurements indicate that combining these three elements—CMX + NIXL + Grove—improves real-time token generation throughput by up to 5x compared to legacy infrastructure.
A 5x gain is impressive. But the direction is what matters even more.
NVIDIA is not merely fabricating faster chips; it is replacing the entire software and hardware stack of AI infrastructure with its proprietary architecture.
7. Vera Rubin NVL72: Turning the Data Center into a Single Chip
While the Vera CPU can be sold as a standalone unit, its true purpose lies elsewhere.
‘Vera Rubin NVL72.’ This name has become one of the most talked-about terms in the 2025 AI infrastructure landscape.
The NVL72 is a single unified rack. Housed within this rack are:
72 Rubin GPUs.
36 Vera CPUs.
BlueField-4 DPUs.
ConnectX-9 SuperNICs.
NVLink 6 switches.
Spectrum-6 Ethernet switches.
Groq LPUs, acquired for $20 billion in December 2025.
A total of seven distinct types of silicon. NVIDIA terms this approach ‘Extreme Co-design.’
Rather than designing individual chips and piecing them together, it means co-architecting the entire rack from scratch as a single unified computer.
The NVLink-C2C interconnect links CPUs and GPUs at 1.8 TB/s—seven times faster than PCIe Gen 6. -> At this bandwidth, data transfer latency between CPU and GPU never leaves the GPU idling.
Taking a closer look at the Rubin GPU: TSMC 3nm process, 336 billion transistors, 288GB HBM4 memory with 22 TB/s bandwidth, and single-chip inference performance of 50 PFLOPS—a 5x leap over Blackwell.
Then comes the Groq LPU. Groq was engineered around a unique design philosophy.
To eliminate the bus bottlenecks caused by GPUs frequently communicating with high-capacity HBM memory, Groq packed the majority of the silicon die with ultra-fast SRAM.
Featuring approximately 500MB of on-chip SRAM, 80 TB/s memory bandwidth, and an architecture driven by a deterministic compiler that pre-schedules the precise timing of every operation, it renders complex hardware-level branch prediction entirely obsolete.
Where does this architecture excel? In the decode phase, where tokens are generated sequentially at lightning speed.
To grasp the inference pipeline within an NVL72 rack, LLM inference must be broken down into two stages:
Prefill: The stage where input prompts are processed all at once. This demands massive parallel matrix compute and is handled by the Rubin GPU.
Decode: The stage where subsequent tokens are generated sequentially one by one after the first token emerges. This is sequential and memory-bandwidth bound, handled by the Groq LPU.
-> Once Rubin finishes the prefill stage, the execution context transitions to the Groq LPU.
Groq’s 80 TB/s SRAM bandwidth then generates tokens at blinding speed.
As a result of this architectural separation, the Vera Rubin NVL72 delivers a 35x improvement in inference output per watt on trillion-parameter models compared to Blackwell.
35 times. Anyone can see this goes far beyond a typical generational upgrade.
8. A $200 Billion Territory Claim
At GTC 2025, Jensen Huang calculated that there are currently roughly 1 billion general-purpose PCs in the world, each with a CPU. Just as human workers require CPUs in their devices, AI agents operating across digital infrastructure require CPUs as well.
According to Huang’s declaration, tens of billions of AI agents will each need CPUs to provide isolated sandboxed computation. If the market structure monopolized by Intel and AMD for decades shifts, the addressable market vacuum is worth $200 billion.
While this might sound like hyperbole, the fact remains that $20 billion in pre-orders arrived in the very first year.
Intel’s 2024 annual revenue for its Data Center and AI group was $12.8 billion. The Vera CPU surpassed that figure in its first year alone.
Yet this raises a more critical question:
Why is NVIDIA making CPUs such an existential threat to Intel and AMD?
NVIDIA is a GPU company. Even if they build CPUs, doesn’t it just add one more competitor to the mix?
There is a fundamental reason why that perception misses the mark.
While NVIDIA’s Vera CPU can be sold independently,
when integrated into the NVL72, it presents an entirely different value proposition.
Using that CPU makes GPU interconnects 7x faster,
optimizes interfacing with CMX,
and allows the DOCA Memos software stack to run seamlessly.
Outside of NVIDIA’s ecosystem, none of these optimization benefits can be realized.
This is the essence of vertical integration. The CPU is not merely an end in itself; it is the wedge that closes off all escape routes.
Until now, hyperscalers had choices.
Even if forced to buy GPUs from NVIDIA, they could still rely on AMD or custom in-house designs for CPUs.
That served as their final bastion of control over infrastructure. Now, NVL72 is taking direct aim at that very fortress.
9. Resistance from Three Fronts
Naturally, this offensive faces simultaneous resistance from three fronts.
The Arm-Meta Alliance
In 2025, Arm CEO Rene Haas jointly announced an ‘AGI CPU’ reference architecture with Meta’s infrastructure team.
(Packing 136 flagship ‘Neoverse V3’ cores into a single package at under 300W, allowing 8,160 physical cores to run in a single standard air-cooled rack.)
The core message: Higher core density can be achieved via open architecture without needing NVIDIA’s complex liquid cooling infrastructure.
CapEx reduction per gigawatt: up to $10 billion.
This is an argument for economic independence rather than pure technical superiority:
proving that agentic orchestration core density demands can be met without vertical lock-in to NVIDIA.
Accelerated In-House Silicon from the Big Three Cloud Providers
Google’s Axion. Amazon’s Graviton 5. Microsoft’s custom silicon.
All are accelerating efforts to handle the orchestration overhead of their services using proprietary in-house silicon rather than commercial merchant CPUs.
Amazon’s Graviton and Nitro hybrid systems have already reached a $20 billion annual revenue run-rate.
Google Axion claims a 65% price-performance improvement and 60% better energy efficiency over legacy solutions.
Their strategy is straightforward:
Leverage the massive scale of their cloud ecosystems to reduce payments to NVIDIA and pass those savings on to cloud customers via lower pricing.
Geopolitical and Regulatory Headwinds
At Computex, Jensen Huang acknowledged that the $200 billion TAM projection includes demand from China.
However, U.S. Department of Commerce export controls continue to block direct export channels of high-end silicon to China.
China is simultaneously pushing its domestic AI ecosystem and hardening policies against importing U.S. semiconductors.
Furthermore, the structure of the $20 billion Groq acquisition sparked controversy.
Allegations that the de facto acquisition—structured via ’licensing agreements and talent acqui-hiring’—intentionally bypassed FTC antitrust merger reviews prompted U.S. Senators Edward Markey and Elizabeth Warren to demand an official investigation.
How much these three headwinds can decelerate NVIDIA’s expansion remains an open question.
10. What Happens When a Monopoly Reaches Completion
History in computing reveals a recurring pattern:
When vertical integration delivers extreme performance,
-> the majority chooses lock-in for the sake of that performance.
-> And once that dependency runs deep enough, the open ecosystem strikes back.
It happened with IBM mainframes. It happened with DEC VAX. It happened with Intel-Microsoft’s Wintel alliance. It happened with Apple’s vertical integration.
In each instance, vertically integrated empires dominated for extended periods. Yet in every case, open ecosystems eventually carved out territory the empire failed to capture—or made too prohibitively expensive.
The sheer scale of what NVIDIA is currently constructing is unprecedented.
Vertically integrating the CPU, GPU, DPU, network switches, memory storage, and software stack all at once is not a monopoly over a single product category—it is a monopoly across the entire layer of computing infrastructure.
Reports have emerged that even inside OpenAI and Anthropic, internal discussions are underway expressing deep concern over locking global inference compute into a single hardware vendor’s ecosystem.
Time will tell whether this is a prescient warning or an unfounded worry. However, the emergence of this concern is itself an important signal that the industry is calculating the cost of dependency before capitulating to raw performance.
NVIDIA is no longer just a company selling GPUs.
It is transforming into the architect of the entire computing infrastructure for the agentic AI era. The Vera CPU is the most explicit declaration of that ambition.
11. Rewriting the Grammar of Infrastructure
Let us take a step back.
What are we witnessing right now?
The NVIDIA Vera CPU is an impressive chip. But that alone does not explain why this story matters. Faster CPUs are released all the time.
This is different because the chip is living proof that the fundamental design principles of AI infrastructure are undergoing a paradigm shift.
The traditional design principle was this:
Add AI accelerators on top of general-purpose CPUs.
Connect them via standard interconnects.
Store data on general-purpose storage.
Each layer is modular and interchangeable.
The new design principle is fundamentally different:
Co-design every layer specifically around the nature of AI workloads.
Prioritize extreme optimization over modular interchangeability.
And concentrate the authority of that optimization under a single provider.
This shift represents a philosophical pivot—the age-old tension between the open-system principles of versatility and modularity versus the vertically integrated principles of optimization and efficiency.
That tension has entered a new chapter in the era of AI.
The performance extremes demanded by AI have reached levels unachievable by general-purpose systems. And we have entered an era where whoever reaches that echelon first claims the entire market.
The Vera CPU is the spearhead toward that horizon.
And far more than whether this specific move succeeds or fails, the very fact that this shift has begun is rewriting the future of computing infrastructure.
12. The World Where Agents Work
Finally, looking from a slightly different perspective:
Imagine the “tens of billions of AI agents” described by Jensen Huang.
Each working inside an isolated sandbox, equipped with dedicated memory, executing tasks via dedicated tools.
Like human knowledge workers, they take on tasks, report findings, and await the next assignment.
What is the CPU in this world? It provides the thinking space for the agent. -> It gives them hands to manipulate tools. It provides the pathway to store and retrieve memory.
Just as human workers need desks and computers, AI agents require CPUs.
NVIDIA did not engineer the Vera CPU merely to fill an empty market niche.
It is a declaration to architect the entire computing infrastructure of the agentic AI era from the ground up.
Where do Intel and AMD stand before this declaration?
What are Arm, Google, and Amazon preparing in response?
And how will the outcome of this struggle dictate the capabilities and economics of AI agents?
That is the true essence of the unspoken war unfolding quietly behind the GPU.
References
- NVIDIA Corporation, “Vera Rubin Platform Architecture Whitepaper” (Jensen Huang, 2025)
- Michael Larabel, “NVIDIA Vera CPU Benchmarks on Linux,” Phoronix (2025)
- Jonathan Ross, “A Software-Defined Tensor Streaming Processor for Large-Scale Machine Learning,” Groq Inc.
- Arm Editorial Team, “Arm Everywhere: Accelerating Agentic AI in Modern Datacenters” (Rene Haas, 2025)
- Vincent Hsu, “IBM Storage Scale integration with NVIDIA Dynamo Serving Engines,” IBM Systems Group (2025)
- NVIDIA Developer Documentation, “DOCA Memos API Reference Guide” (2025)
- NVIDIA Research, “Agentic AI Orchestration Bottleneck Analysis: CPU vs GPU Latency Distribution” (2024)
- Tom’s Hardware, “NVIDIA Vera CPU vs Intel Xeon 6980P vs AMD EPYC 9575F: Benchmark Comparison” (2025)
- IEEE Computer Architecture Letters, “Spatial Multithreading for Agentic Workloads: Design Principles and Performance Analysis” (2025)
- Semiconductor Engineering, “SOCAMM2 Form Factor: Enabling LPDDR5X in Server Environments” (2025)
- Next Platform, “Understanding NVIDIA’s CMX Context Memory Extension Architecture” (2025)
- Hot Chips 37, “NVIDIA Olympus Microarchitecture Deep Dive” (2025)
- Solidigm Technical Brief, “D7-PS1010 NVMe SSD for AI Inference KV Cache Workloads” (2025)
- Reuters, “U.S. Senators Request FTC Investigation into NVIDIA-Groq Acquisition Structure” (2025)
- Dylan Patel, “The Agentic AI Infrastructure Wars: CPU, Memory, and the Battle for the Control Plane,” SemiAnalysis (2025)