google splits eighth-generation TPU into training and inference chips as AI agents reshape hardware economics

google splits eighth-generation TPU into training and inference chips as AI agents reshape hardware economics

google has split its eighth-generation TPU into two specialized chips, TPU 8t for training and TPU 8i for inference, reshaping how engineering and FinOps teams plan capacity, match models to multi-step agent workloads, and track serving costs per completed task.

prefer designboom on google

[FEATURED]

ONE CHIP, TWO JOBS

 

Agentic workloads broke the shared GPU assumption. Google’s decision to split its eighth-generation TPU into one chip for training and one for inference shows how AI agents are changing the economics of the hardware underneath them.

google splits eighth-generation TPU into training and inference chips as AI agents reshape hardware economics - 1
all images ©Cloudfresh

 

 

GOOGLE SPLITS TPU INTO TRAINING AND INFERENCE SILICON

 

A shared GPU fleet does two different jobs. The same chips spend weeks training a model, then answer user requests once that model ships, so clusters, budgets, and roadmaps get sized around one pool of hardware. That setup is starting to split, and teams running models in production will feel it first in capacity planning and cost reviews, the work a Google Cloud partner such as Cloudfresh covers through its Google Cloud Consulting Services. At Google Cloud Next in April 2026, Google announced an eighth TPU generation that, for the first time, includes two distinct chips: TPU 8t for training and TPU 8i for inference. Google says it anticipated rising inference demand several years ago, as frontier models moved into production, and concluded that agents would benefit from chips specialized separately for training and serving.

agentic workloads broke the shared gpu assumption 2

 

 

WHY AGENTS STRAIN A GENERAL-PURPOSE ACCELERATOR

 

Training rewards throughput: the more data a chip processes per second, the sooner the model is ready. Agents load hardware differently. In Google’s description, agentic models reason through problems, run multi-step workflows, and learn from their own actions in continuous loops. Each step waits for the one before it, so memory access and latency start to matter as much as raw compute. TPU 8i is designed around that problem. It pairs 288 GB of high-bandwidth memory with 384 MB of on-chip SRAM, three times the SRAM of the previous generation, to keep a model’s active working set on the chip. Google’s stated reason is that interactions between agents at scale magnify even small inefficiencies. Google also notes that both chips can run either kind of workload, with specialization delivering the efficiency gains. General-purpose hardware still runs agents. Each team now faces the decision of how much efficiency it gives up by keeping one shared fleet for both jobs.

agentic workloads broke the shared gpu assumption 3

 

 

WHAT THE SPLIT DOES TO THE COST LINE

 

Google’s own figures put a size on that efficiency. The company says TPU 8i delivers 80% better performance per dollar than the previous generation, enough to serve nearly twice the customer volume at the same cost, and that both new chips reach up to twice the performance per watt of Ironwood. Power efficiency carries weight because, according to Google, power has joined chip supply as a binding constraint in today’s data centers. For FinOps teams, the practical change is the unit of measurement. Once training and serving run on hardware with different price curves, a single blended line for accelerator hours hides both. Training spend behaves like a project with a start and an end. Serving spend grows with every request an agent handles. Splitting them into two budgets, and tracking serving as cost per completed task, shows which agents justify their compute.

agentic workloads broke the shared gpu assumption 4

 

 

WHAT CHANGES FOR TEAMS RUNNING REASONING PIPELINES

 

The software around an agent decides how efficiently it uses whatever chip sits underneath, and Google rebuilt that layer as well. Vertex AI became the Gemini Enterprise Agent Platform, with an Agent Runtime that supports long-running agents holding state for days, plus Agent Evaluation and Agent Observability, which trace an agent’s reasoning step by step in production. Those traces can show where an agent loops, retries, or calls a larger model than a task needs, which is where serving cost builds up. Model choice is the second factor teams control. The platform offers more than 200 models through Model Garden, including Gemini 3.1 Pro, the open Gemma 4, and Anthropic’s Claude models, so each step of a pipeline can run on a model sized for it.

 

According to Oleh Maksymovych, Co-Founder, General Manager, Google Cloud & Data Analytics Expert at Cloudfresh, the order of decisions matters: ‘Once an agent is in production, the question shifts from how to train a model to what each answer costs. That’s why we start with the workload: which tasks need a large model, which run well on a smaller one, and how often each is called. The hardware decision comes after that.’ A Cloudfresh project on Google Cloud shows the approach. A product company with more than 80 million users across dating and women’s mental health apps first built several AI scenarios on open Gemma models deployed on rented GPUs, and later through external AI providers. As the number of use cases grew, the team needed more stability, scalability, and control in production. After the move to Gemini, with specific models selected for specific tasks and request costs optimized for production load, profile review coverage rose from 17% to 100% without raising costs or resource demands, according to the company, and about 80% of profiles are approved automatically in under five seconds. Lock-in is a fair concern with any custom chip. Google lists native support for JAX, MaxText, PyTorch, SGLang and vLLM on both TPU 8 platforms, so teams serving models with those frameworks can evaluate TPU 8i on the tools they already use.

agentic workloads broke the shared gpu assumption 5

 

 

WHERE TO START

 

Teams can prepare on the hardware they already run. Tag every accelerator workload as training or serving, attach serving cost to the agent and task that generated it, and flag tasks that call a larger model than their output requires. With that map in place, moving serving onto inference-specific hardware becomes a calculation per agent, based on the team’s own numbers.

agentic workloads broke the shared gpu assumption 6

google-tpu-chips-cloudfresh-google-cloud-consulting-services-designboom-1800-2

agentic workloads broke the shared gpu assumption 7

agentic workloads broke the shared gpu assumption 8

google-tpu-chips-cloudfresh-google-cloud-consulting-services-designboom-1800-3

agentic workloads broke the shared gpu assumption 9

agentic workloads broke the shared gpu assumption 10

agentic workloads broke the shared gpu assumption 11

agentic workloads broke the shared gpu assumption 12

 

project info:

 

name: Google Cloud Consulting Services

cloud consulting partner: cloudfresh | @cloudfresh_com

 

 

designboom has received this project from our DIY submissions feature, where we welcome our readers to submit their own work for publication. See more project submissions from our readers here.