Success Stories

The model store devours the expensive TPU hour

In GKE with TPUs, model loading is the most expensive moment during AI inference. Why cold starts and host memory drive up the cloud bill.

By Alec Chizhik July 11, 2026 5 min read

The model store devours the expensive TPU hour

On a TPU VM with four chips, every minute costs money. When Google made a 449-gigabyte model ready for inference after more than ten minutes in a documented benchmark, the node paid for compute power that hadn’t even started calculating yet. That very first moment-when the model loads-decides the real cloud bill for AI inference on Google Kubernetes Engine.

Key Takeaways

The loading phase is the cost driver. For a 480-billion-parameter model, loading time on a TPU dropped from over 630 seconds to under 280 seconds once weights were streamed directly from object storage instead of taking the local detour.
The host memory requirement halves. The classic load path peaked at 881 gigabytes of host RAM; streaming managed with 436. The difference is budget that a node pool would otherwise have to reserve indefinitely.
The root cause lies in the architecture. TPU nodes have no local SSDs. The classic route temporarily consumes twice the model size in RAM. Ignoring this means paying for over-provisioning and sluggish scaling.

Why the Expensive TPU Waits Before It Even Computes

An accelerator costs per hour whether it’s generating tokens or merely shoving a model into memory. With small models the overhead is invisible. With a model whose weights span hundreds of gigabytes, the loading step becomes the longest phase in a pod’s life cycle.

That directly undermines autoscaling. When a cluster scales up during peak load, every new pod must finish loading its model before it can answer the first request. If that takes several minutes, a dilemma appears: either scaling reacts too slowly and users wait, or the team keeps expensive spare capacity permanently warm. Both paths burn TPU hours.

The math is uncomfortable. A node that loads for ten minutes and then runs for an hour sacrifices roughly one-seventh of its paid runtime to a process that produces not a single token.

The Hidden Memory Surcharge at Startup

The classic TPU load path routes through the host’s RAM. The model is first read entirely into CPU memory, split there for the chips, and only then transferred. For PyTorch models without specialized load logic, this creates a memory spike of about twice the model size because the checkpoint and the prepared copy briefly coexist.

That double-booking is the real cost center. A node pool must be sized for the spike, not normal operation. Budget is reserved for a moment that occurs only at launch.

// Memory spike at load

881 GB → 436 GB

Host-RAM requirement for a 449-GB model: classic load path versus direct streaming from object storage.

TPU nodes without local SSDs force a decision

Unlike many GPU instances, TPU nodes lack a fast local disk from which a model could be loaded. The weights must come from object storage or attached volumes. This intensifies the trade-off between load time, storage costs, and the risk that a node will exhaust its RAM on startup.

The workaround Google documents for this scenario is the open-source Run:ai Model Streamer. It bypasses the local detour and streams the weights in parallel from object storage straight into the chip’s memory. For the general case, the documentation cites up to six-times faster load times compared with conventional methods; the TPU case measured here delivered roughly a two-times improvement. TPU support in vLLM begins with version 0.18.0.

Key context: the measured two-times jump applies to the described scenario of large PyTorch models. Models that already use optimized loading logic see smaller gains. If you plan to adopt the streamer, validate it against your own model type rather than copying the benchmark figure.

Which levers platform teams can actually pull

The first lever is the loading path itself. Streaming from object storage lowers the memory spike and enables smaller, cheaper node pools-two wins at once: faster pod readiness for scaling and less reserved RAM per node.

The second lever is caching. A persistent compilation cache in object storage cuts subsequent startups noticeably because the expensive preparation step no longer runs for every pod. For zonal acceleration, a read cache can be placed in front of object storage.

The third lever is planning. True scale-to-zero remains costly on TPUs because every cold start still pays the full load penalty. For fluctuating workloads, selective elasticity with pre-warmed reserve pods is often cheaper than pure up-and-down scaling. If you take your AI-infrastructure costs seriously-say, at the Google Cloud Summit DACH 2026-this calculation cannot be ignored.

Frequently Asked Questions

What is the Run:ai Model Streamer?

The Run:ai Model Streamer is an open-source component that loads model weights in parallel from an object store such as Cloud Storage directly into an accelerator’s memory. It skips the local disk and the detour through host RAM, letting large models start faster and with a lower memory peak.

Why is cold-start on TPU inference a cost issue?

An accelerator costs per hour even while it loads a model. If loading a large model takes several minutes, scaling is delayed and teams either react slowly or keep expensive reserve capacity on standby-both of which raise the cost per successfully answered request.

From which vLLM version does the streamer run on TPU?

TPU support requires vLLM 0.18.0 or newer. For GPUs the streamer can already connect to Cloud Storage from version 0.11.1; it is enabled via an additional flag in the launch command.

Does the two-times speed-up apply to every model?

No. The measured halving of load time and memory peak refers to large PyTorch models that follow the classic path through host memory. Models with already incremental loading logic benefit less. Before switching, test with your own model.

How does model load time relate to autoscaling?

Every newly launched pod must load its model before it can answer requests. A long load phase makes auto-scaling sluggish during traffic spikes. Shorter load times let pods become ready faster, reduce the need for pre-warmed reserve capacity, and improve the utilization of paid accelerators.

Editor’s Reading Picks

Higher PUE limits and longer deadlines: how the EnEfG amendment relaxes data-centre obligations
AI integration goes enterprise-ready: the Model Context Protocol under the Linux Foundation
Near top-tier performance, cheaper and trained across three continents