Introduction: The Inference Inflection Point

AI adoption is reaching a critical juncture. The industry's focus is rapidly shifting from training massive models to deploying them at scale—inference. For AI chip startups vying for a slice of Nvidia's dominant market share, this transition represents a second chance to carve out a niche. But the window is narrowing. Nvidia's $20 billion acquihire of Groq in December 2025 signals that the inference market is already consolidating. This briefing analyzes the strategic consequences of this shift, identifies winners and losers, and provides actionable intelligence for executives navigating the new landscape.

Context: What Happened

Inference workloads are fundamentally different from training. They require a diverse mix of compute, memory, and bandwidth, depending on the application—large batch processing, real-time AI assistants, or code generation. This heterogeneity has led to a trend toward disaggregated inference, where different parts of the inference pipeline (prefill and decode) are handled by specialized hardware. Nvidia's Groq acquisition is a prime example: Groq's SRAM-heavy LPUs excel at fast token generation (decode), while Nvidia's GPUs handle compute-heavy prefill. AWS followed suit, announcing a platform using its Trainium accelerators for prefill and Cerebras Systems' wafer-scale chips for decode. Intel also unveiled a reference design pairing its GPUs with SambaNova's RDUs. Meanwhile, startups like Lumai and Tenstorrent are pursuing alternative paths—optical computing and RISC-V-based general-purpose platforms, respectively.

Strategic Analysis: The Disaggregation Opportunity

Why Inference Favors Specialization

Inference is not a monolithic workload. Large batch inference demands high compute throughput, while real-time applications require low latency. Disaggregation allows each phase to be optimized independently. For startups, this creates entry points where they can outperform GPUs on specific metrics. Groq's LPUs, for instance, achieved record-breaking token generation speeds, albeit at the cost of scalability. Nvidia's integration of Groq's technology into its ecosystem validates the disaggregation approach. However, it also raises the bar for other startups: they must either partner with hyperscalers or differentiate enough to survive independently.

Optical Computing: A Dark Horse?

Lumai's optical inference accelerator represents a radical departure. By using light instead of electrons for matrix multiplication, Lumai claims it can achieve an exaOPS of AI performance within a 10kW power budget by 2029. If realized, this would be a game-changer for energy-constrained data centers. However, the technology is still nascent—currently capable of running only billion-parameter models like Llama 3.1 8B or 70B. Lumai's strategy to first target compute-bound batch inference and later expand to prefill is prudent, but the 2029 timeline means it will face intense competition from established players and other startups that may scale faster.

Tenstorrent's Counter-Narrative: Simplicity Over Complexity

Not everyone is betting on disaggregation. Tenstorrent's CEO Jim Keller explicitly criticized the trend, calling it a 'complex solution unlikely to be compatible with changes in AI models.' Instead, Tenstorrent's RISC-V-based Galaxy Blackhole platform aims to be a general-purpose inference accelerator. This approach appeals to customers wary of vendor lock-in and eager for open ecosystems. However, it also means Tenstorrent must compete head-on with Nvidia's GPUs on performance and software ecosystem. Keller's track record (leading Apple's A4/A5, AMD's Zen, and Tesla's Autopilot hardware) lends credibility, but execution risk remains high.

Winners & Losers

Winners

  • Nvidia: The Groq acquihire strengthens its inference IP and talent pool, reinforcing its dominance across the AI stack.
  • AWS: By integrating Trainium with Cerebras, AWS offers a differentiated, cost-optimized inference solution that strengthens its cloud AI portfolio.
  • Lumai: Optical computing could capture the high-efficiency segment if it meets its 2029 targets.
  • Tenstorrent: RISC-V and open-source appeal may attract hyperscalers seeking alternatives to Nvidia.

Losers

  • Traditional CPU/GPU vendors (Intel, AMD): Disaggregation and specialized chips erode their general-purpose inference market share.
  • Small inference startups without differentiation: Consolidation and hyperscaler in-house chips squeeze them out.
  • Proprietary hardware vendors (e.g., Graphcore): Open ecosystems and disaggregation reduce lock-in advantages.

Second-Order Effects

The disaggregation trend will accelerate software stack fragmentation, as each combination of prefill and decode hardware requires optimized compilers and runtimes. This could increase switching costs for customers, paradoxically benefiting Nvidia if its CUDA ecosystem becomes the common denominator. Additionally, the rise of optical computing may spur investment in photonic interconnects and packaging, reshaping the semiconductor supply chain. Finally, Tenstorrent's RISC-V push could catalyze a broader open-source hardware movement, challenging proprietary architectures across the board.

Market / Industry Impact

The inference chip market is projected to grow from $15 billion in 2025 to over $80 billion by 2030. Disaggregation will create multiple sub-markets: prefill accelerators, decode accelerators, and integrated solutions. Startups that can secure partnerships with hyperscalers (like Cerebras with AWS) will gain credibility and scale. However, Nvidia's aggressive M&A and R&D spending mean it will likely capture the largest share. The real battleground will be in power efficiency and total cost of ownership, where optical and analog approaches could disrupt.

Executive Action

  • Evaluate disaggregation for your inference workloads: Assess whether splitting prefill and decode can reduce costs or latency. Pilot with hyperscaler offerings like AWS's Trainium-Cerebras platform.
  • Monitor optical computing progress: Track Lumai and other photonic startups. If they hit their 2029 targets, they could reshape data center economics.
  • Diversify vendor risk: Consider open-source alternatives like Tenstorrent's RISC-V to avoid over-reliance on Nvidia. Engage with the RISC-V ecosystem now to influence software tooling.

Why This Matters

The inference shift is not just a technological evolution—it's a strategic inflection point. Companies that bet on the right architecture now will gain a multi-year cost and performance advantage. Those that ignore the disaggregation trend risk being locked into suboptimal solutions as AI deployment scales exponentially.

Final Take

Nvidia's $20B Groq bet underscores that inference is the next frontier. But the diversity of workloads means there is room for multiple winners—if they execute flawlessly. Startups like Lumai and Tenstorrent offer genuine alternatives, but they face an uphill battle against Nvidia's ecosystem and hyperscaler in-house chips. The next 12 months will separate the contenders from the pretenders. Executives should act now to position their organizations for the disaggregated future.



Source: The Register

Rate the Intelligence Signal

Intelligence FAQ

As AI models are deployed at scale, inference workloads dominate compute demand. Unlike training, inference is heterogeneous, allowing specialized chips to outperform GPUs on specific tasks.

It validates disaggregated inference and gives Nvidia a leading position in low-latency decode. It also raises the bar for startups, which must now either partner with hyperscalers or differentiate radically.

Increased system complexity, software fragmentation, and potential incompatibility with future model architectures. Tenstorrent's Jim Keller argues that simpler, general-purpose designs may be more sustainable.