Small Models Devour Large GPU Budgets through Preallocation
vLLM-Prealloc and process-per-model devour cards. Shared Inference and Memory Scheduling save euros before the next GPU order.
Small models should cut costs. In practice, four vLLM instances often occupy four GPUs because each engine pre-allocates most of the memory. The bottleneck isn’t the card-it’s the serving stack.
Key takeaways
- The pre-allocation trap. Many inference engines reserve most of the GPU memory at startup and don’t share it fairly with neighboring processes.
- Packing beats buying. Running embeddings, rerankers, extractors, and generators on a single card saves on rental costs-if the scheduler dynamically shifts memory instead of demanding four full cards.
- Measure before scaling. Tokens per euro and cold-start seconds are what matter-not the number of model names in the catalog.
Related:Loading the model eats up the expensive TPU hour / Kimi K3: When AI builds its own infrastructure
Four models, four cards, one flawed assumption
FinOps reports then show rising GPU costs despite “smaller” models. The narrative in the steering committee sounds paradoxical. In the node metrics, it’s painfully simple: four processes, four allocations, four bills. Without packing discipline, model specialization becomes a cost driver instead of a savings lever.
Teams break down the agent stack into specialists: embedding, reranking, structured extraction, generation. Each component is small. Yet every vLLM or TEI instance behaves as if it’s the sole owner of the GPU. Allocating 90% of VRAM at startup leaves no room for neighbors. The result? Hardware multiplication-despite “small models.”
Serverless inference only appears to solve this. Cold starts lasting dozens of seconds-sometimes nearing a minute-are deadly for rerankers in search pipelines. Warm pools cost idle time. On-prem or reserved GPUs without shared serving are just expensive versions of the same problem.
What shared inference truly requires
A server process that handles multiple models. Load on first use. LRU eviction when memory gets tight. No “one process per model” dogma. Open-source projects and commercial inference routers are targeting exactly this. The project name doesn’t matter-the architectural question remains: Who owns the memory scheduler?
Platform teams should lock in three SLOs. First, p95 latency per tool type. Second, VRAM headroom under load. Third, cost per 1 million tokens across the mixed stack-not per individual model in ideal conditions. Without these three metrics, every GPU purchase is just a gut decision.
Operational Rules That Save Money
Separate prefill-heavy and decode-heavy workloads if your hardware can handle it. Use cached models for hot paths, lazy loading for peripheral tools. Avoid silent shadow deployments of two full engines “just for testing” on production GPUs. Ensure observability on model-load events: every load is a cost and latency spike.
Shared on-box serving beats the cloud API path when traffic is stable and data stays local. Shared serving outperforms single-model servers as soon as more than two models are consistently hot. A well-defined inference daemon with clear quotas wins against naive Kubernetes-per-pod-per-model setups.
Kubernetes often makes the problem worse when every inference variant runs as its own deployment with a GPU request of 1. The scheduler sees resource requests, not model overlap. Without device-plugin strategies and targeted packing, idle VRAM islands form next to waiting pods.
A pragmatic middle ground for many DACH teams: an inference node pool with a fixed shared server per node, fronted by a lightweight router. Autoscaling at the node level, not per model pod. High-QPS models can stay dedicated. Everything else shares resources.
Document the worst-case scenario. What happens if the generator model displaces the reranker? Is the LRU path acceptable, or do you need pinning for the top two models? Without a pinning policy, the first traffic spike turns into latency chaos.
Monday Checklist
List all inference processes and their VRAM claims. Look for duplicate base weights. Simulate the scenario “reranker + generator simultaneously.” If that burns two GPUs, you don’t have a capacity problem-you have a packing problem. Fix it before placing the next order.
Frequently Asked Questions
Why aren’t small models enough on their own?
Because the serving stack often reserves most of the GPU per process. Model size and reserved memory are decoupled.
Is serverless the solution?
Only if cold starts are acceptable. For synchronous search and agent paths, often not-unless you’re using expensive warm pools.
What should I measure first?
Concurrent models, VRAM claim per process, p95 latency, and cost per token across your real workload mix.
Alternative to the cloud API path?
An option for residual or cost-stable workloads. Not an automatic choice. Shared serving is an operational model with measurable trade-offs.
Editor’s Picks
cloudmagazinLoading the model eats up the expensive TPU hourcloudmagazinPlatform or facade? The honest truth about Platform EngineeringMore from the MBF Media Network
MyBusinessFutureCheap AI from China: What procurement needs to checkDigital ChiefsWashington calls the shots on which AI can run hereSecurityTodayHugging Face breach: Alarms sounded, triage failedImage source: AI-generated (July 2026)
