Google just bet its inference future on a chip built for one model

The race to make AI inference cheaper is pushing chip design beyond general-purpose accelerators. We’re now moving toward silicon that The post Google just bet its inference future on a chip built for one model appeared first on The New Stack .

Google just bet its inference future on a chip built for one model

The race to make AI inference cheaper is pushing chip design beyond general-purpose accelerators. We’re now moving toward silicon that is tailored to individual models. And Google’s reported “Frozen v2” project suggests the Gemini AI model is part of the movement.

First reported by The Information, the unannounced chip would reportedly hardwire parts of Gemini’s architecture while leaving its weights updatable. That compromise could give Google much of the efficiency of model-specific silicon without making the hardware obsolete every time Gemini changes.

A spokesperson for Google tells The New Stack, “Our teams are constantly researching and experimenting with new innovations to deliver maximum performance and efficiency for our users and customers.”

A spokesperson for Google tells The New Stack, “Our teams are constantly researching and experimenting with new innovations to deliver maximum performance and efficiency for our users and customers. While not every project moves into production, this rigorous exploration is central to our full stack approach. By co-designing our hardware and software from the ground up, we ensure our systems are integrated and highly optimized for real-world workloads.”

According to the reporting, Google hopes the chip will help relieve the AI compute crunch that’s made it harder for cloud providers to keep up with demand, while also making Gemini much cheaper and more efficient to serve. Internal projections reportedly estimate the design could deliver six to ten times more tokens per watt than Google’s current generation of AI chips.

If the project moves forward, it would represent a different approach to AI infrastructure because Google would be building hardware specifically for Gemini. For developers, that’s an early indication that future AI systems may be designed with much tighter integration between the model and the hardware beneath it.

Specialized silicon replaces flexibility

Currently, the vast majority of AI inferences run on Nvidia GPUs or Google’s own Tensor Processing Units (TPUs). Because these chips are intended to accommodate a wide variety of AI models, they carry a high degree of processing load.

There’s precedent for this kind of shift. Bitcoin mining followed a similar path, starting with CPUs, then GPUs, before ultimately settling on ASICs. AI inference may be headed in the same direction.

Training still benefits from the flexibility of GPUs, but once a model reaches production, the priority shifts to serving as many requests as possible while using less power. Training still benefits from the flexibility of GPUs, but once a model reaches production, the priority shifts to serving as many requests as possible while using less power.

Competitors hardwire their own

Google isn’t the only company looking beyond general-purpose GPUs for AI inference. As inference becomes a larger share of AI workloads, more companies are experimenting with specialized hardware designed to improve performance while using less power.

Even Nvidia, whose GPUs dominate the AI market, has invested heavily in inference, striking a reported $20 billion deal with Groq last year to license the startup’s technology.

Among the more ambitious efforts is Canadian startup Taalas. The company has demonstrated a chip with an entire 8-billion-parameter Llama model embedded directly into the silicon. By keeping the model on the chip instead of constantly moving data back and forth from external memory, Taalas says it can dramatically speed up inference, claiming throughput of roughly 17,000 tokens per second.

Meanwhile, other companies are optimizing the spatial distance between memory and compute to avoid hardware lock-in. d-Matrix’s new Corsair platform uses an SRAM-based in-memory compute architecture rather than relying on standard high-bandwidth memory (HBM) packaging. Similarly, SambaNova is deploying custom dataflow technology with a three-tier memory architecture to maximize tokens per watt for advanced workflows.

Google’s Frozen v2 sits uniquely in this landscape - borrowing the hyper-efficiency of hardwired architectures like Taalas, but retaining just enough flexibility to remain viable across multiple product cycles.

Freezing architecture, not weights

The “Frozen” name reportedly comes from the idea of permanently etching part of Gemini’s design into the chip itself. According to The Information, Google’s engineers have spent years trying to find the right balance between efficiency and flexibility.

An earlier concept, led by Google DeepMind Chief Scientist Jeff Dean, reportedly would have embedded Gemini’s model weights directly into the silicon. That idea was eventually abandoned because it would have tied the hardware to a single version of the model, dramatically limiting its useful life as Gemini continued to evolve.

Frozen v2 reportedly takes a different approach. Rather than locking in the weights, the chip would hardwire parts of Gemini’s underlying design while still allowing the weights to be updated over time. That would let Google continue improving the model without needing to replace the hardware every time Gemini receives a new version.

Rather than locking in the weights, the chip would hardwire parts of Gemini’s underlying design while still allowing the weights to be updated over time.

Cheaper inference reaches developers

By moving some of Gemini’s execution into the chip itself, Google could reduce some of the overhead that comes with running models on more general-purpose hardware. That could translate into lower latency, especially for applications that depend on near real-time responses.

Google’s reported goal of delivering six to ten times more tokens per watt is ultimately about efficiency, and those savings could eventually make their way to enterprise teams building on Gemini, through lower API costs or more available capacity. While Google hasn’t said how it would pass those gains along, reducing the cost of inference has become a priority across the industry.