Success Stories

The model store devours the expensive TPU hour

In GKE with TPUs, model loading is the most expensive moment during AI inference. Why cold starts and host memory drive up the cloud bill.

By Alec Chizhik July 11, 2026 5 min read

The model store devours the expensive TPU hour

On a TPU VM with four chips, every minute costs money. When Google made a 449-gigabyte model ready for inference only after more than ten minutes in a documented benchmark, the node paid for compute power that hadn’t even started working yet. It’s this very start-up moment that decides the real cloud bill for AI inference on Google Kubernetes Engine.

Key Takeaways

The loading phase is the real cost driver. For a 480-billion-parameter model, loading time on a TPU dropped from over 630 to under 280 seconds once weights were streamed directly from object storage instead of taking the local detour.
Host memory demand is cut in half. The classic load path peaked at 881 GB of host RAM; streaming managed with 436 GB. That difference is reserved budget a node pool otherwise has to keep permanently on ice.
The root cause lies in the architecture. TPU nodes have no local SSDs. The traditional load path temporarily consumes twice the model size in RAM. Ignoring this means paying for over-provisioning and sluggish scaling.

Why the Expensive TPU Sits Idle Before It Even Starts Computing

An accelerator costs per hour whether it’s generating tokens or merely shoving a model into memory. With small models the overhead is invisible. With a model whose weights run into the hundreds of gigabytes, the loading process becomes the longest phase in a pod’s life cycle.

This directly hits auto-scaling. When a cluster scales up during a traffic spike, every new pod must finish loading its model before it can answer the first request. If that takes several minutes, a dilemma appears: either the scaling reacts too late and users wait, or the team keeps expensive spare capacity permanently warm. Both paths burn through TPU hours.

The math is uncomfortable. A node that spends ten minutes loading and then works for an hour loses roughly one seventh of its paid runtime to a process that produces not a single token.

The Hidden Memory Surcharge at Start-up

The classic load path on a TPU goes via the host RAM. The model is first read entirely into CPU memory, split there for the chips, and only then transferred. For PyTorch models without specialized load logic, this creates a memory spike of about twice the model size because the checkpoint and the prepared copy briefly coexist.

That dual occupancy is the real cost driver. A node pool must be sized to survive the peak, not the steady state. Capacity is reserved for a moment that occurs only at start-up.

// Memory spike at load time

881 GB → 436 GB

Host RAM demand for a 449 GB model: classic load path versus direct streaming from object storage.

TPU nodes without local SSDs force a decision

Unlike many GPU instances, TPU nodes lack a fast local disk from which a model could be loaded. The weights must come from object storage or attached volumes. This intensifies the trade-off between load time, storage costs, and the risk that a node will overflow its RAM on startup.

The workaround Google documents for this scenario is the open-source Run:ai Model Streamer. It bypasses the local detour and pushes the weights in parallel directly from object storage into chip memory. For the general case, the documentation cites up to six-times faster load times compared with conventional methods; the TPU case measured here came in at roughly a two-times improvement. TPU support in vLLM starts with version 0.18.0.

Important for context: the measured jump to a two-times factor applies to the described case of large PyTorch models. For models that already use optimized loading logic, the gain is smaller. If you plan to deploy the streamer, test it with your own model type rather than adopting the benchmark figure.

Which levers platform teams actually have

The first lever is the loading path itself. Streaming from object storage lowers the memory spike and allows smaller, cheaper node pools – two immediate benefits: faster pod readiness for scaling and less reserved RAM per node.

The second lever is caching. A persistent compilation cache in object storage cuts subsequent startups noticeably because the expensive preparation step no longer runs for every pod. For zonal acceleration, a read cache can be placed in front of object storage.

The third lever is planning. True scale-to-zero remains costly on TPUs because every cold start pays the full loading cost. For fluctuating workloads, selective elasticity with pre-warmed reserve pods is often cheaper than pure up-and-down scaling. If you take the operating costs of AI infrastructure seriously – say, at the Google Cloud Summit DACH 2026 – this calculation cannot be ignored.

Frequently Asked Questions

What is the Run:ai Model Streamer?

The Run:ai Model Streamer is an open-source component that loads model weights in parallel from an object store such as Cloud Storage directly into accelerator memory. It skips the local disk and the detour through host RAM, letting large models start faster with a lower memory spike.

Why is cold-start inference on TPUs a cost issue?

An accelerator costs per hour even while it is loading a model. If large models take several minutes to load, scaling is delayed and teams must either accept slow responses or pay for expensive reserve capacity – both of which raise the cost per successfully answered request.

From which vLLM version does the streamer run on TPU?

For the TPU path, vLLM 0.18.0 or later is required. For GPUs, the streamer can already connect to Cloud Storage from version 0.11.1; it is enabled via an additional flag in the launch command.

Does the two-times speed-up apply to every model?

No. The measured halving of load time and memory spike refers to large PyTorch models that follow the classic path through host memory. Models with already incremental loading logic see smaller gains. Run a test with your own model before switching.

How does model load time relate to autoscaling?

Every newly started pod must load its model before it can answer requests. Long load phases make automatic scaling sluggish during traffic spikes. Shorter load times get pods ready faster, reduce the need for pre-warmed reserve capacity, and improve the utilization of paid accelerators.

Editor’s Reading Picks

Higher PUE thresholds and extended deadlines: How the EnEfG amendment relaxes data-centre obligations
AI integration goes enterprise-ready: The Model Context Protocol under the Linux Foundation
Near top-tier performance, cheaper and trained across three continents