Downloadable Doesn’t Mean Deployable

In early August, rankings of the best open AI models circulate, with one model leading the pack despite its weights not yet being released. For IT managers and cloud architects, this raises a question no feed can answer: What criteria should guide the decision for or against running AI locally?

Key Takeaways

Related:Soofi S: “Sovereign” Doesn’t Mean “Winner”  /  Inkling: What the New Open-Weights Model Can Do

The Debate Outpaces Availability

Alibaba unveiled Qwen3.8-Max in early August: 2.4 trillion parameters, 95 billion active, Mixture-of-Experts architecture, and a context window of up to one million tokens. Alongside the flagship model, a second checkpoint – Qwen3.8-27B – was slated for open release, though its weights had not yet been published at the time of the announcement; the company cited the following week for their availability.

This did not stop rankings or explainer videos from treating both models as already available. Community experts quickly corrected the record, sometimes bluntly. Yet the pattern remains structural rather than a one-off error: announcements, benchmark tables, and community assessments emerge within hours, while weights – and operational experience – take weeks to materialize.

Procurement teams aligning their plans to this tempo risk betting on a model that doesn’t yet exist – or one that will never run in their own environment.

Open Weights, closed by hardware

One phrase from the debate captures the essence and has become a catchphrase: “Open Weights, closed by hardware.” When running a model demands hundreds of thousands of GPUs, it’s no longer a local model. It’s a cloud model with downloadable weights.

The scale bears this out. At the end of July, Moonshot released the weights for Kimi K3 – 2.8 trillion parameters with a one-million-token context. In the community, a half-joking prep list circulated suggesting you first secure a hardware budget in the high six-figure range. The sobering counterpoint: most open models simply can’t be run locally because the resources simply aren’t there.

There are exceptions – impressive ones. With aggressive 1-bit quantization, Kimi K3 can be squeezed from roughly 1.56 terabytes down to about 594 gigabytes, a reduction of more than 60 percent while retaining roughly 79 percent of its original accuracy. That will run on a well-equipped workstation. But it remains a special case: the loss in accuracy is real, and the memory ceiling is merely shifted, not eliminated.

A quick calculation puts the scale in perspective. 2.4 trillion parameters at 4 bits require about 1.2 terabytes just for the weights. An NVIDIA H200 has 141 gigabytes. That’s nine GPUs before a single byte of KV cache is allocated. With a one-million-token context, that cache becomes substantial. “Locally” here means your own data center or a rented cluster – not the workstation in the next room.

Active parameters don’t dictate memory needs

The sticking point where procurement decisions fail: with Mixture-of-Experts models, the memory footprint is driven not by the number of active parameters. DeepSeek V4 Flash has 13 billion active parameters yet loads all 284 billion into memory. If you read “13B active” and order a 16-gigabyte card, you’re off by an order of magnitude.

This pattern runs through the current generation of models. Qwen3.8-Max: 2.4 trillion total, 95 billion active. Zhipu’s GLM-5.2: 744 billion total, about 40 billion active, MIT-licensed on Hugging Face. DeepSeek V4 Pro: 1.6 trillion total, 49 billion active, also MIT. Active parameters explain the compute cost per token; memory demand follows the total size.

Real-world measurements from the community reveal the practical ceiling. These are individual benchmarks, not vendor specs, but they align with one another: DeepSeek V4 Flash, with optimized inference software, can push throughput from roughly 3 to nearly 26 tokens per second on a single consumer card with 24 gigabytes. Sebastian Raschka reports around 40 tokens per second for 30-billion-parameter MoE models on a well-specced workstation and notes that an 118-billion-parameter model with 8 billion active parameters stays under 80 gigabytes of RAM.

Rule-of-thumb for 4-bit quantization: 7 billion parameters fit in 8 gigabytes, 14 billion in 12, the 24-to-32-billion bracket needs 16 to 24, and a 70-billion model starts at 40 gigabytes. These are the working sizes for day-to-day use – not the headline-grabbing behemoths.

What was top-tier yesterday runs today on a single card

Memory physics is one half of the story. The other is the speed at which the boundary shifts. It’s the real news for infrastructure planning.

One example with hard numbers: In April 2026, Alibaba released Qwen3.6-27B, a dense model with 27 billion parameters that runs on a single RTX 4090 or a Mac with 24 gigabytes of RAM. On SWE-bench Verified, one of the tougher benchmarks for agentic programming, it scores 77.2 percent – beating its own previous flagship model with 397 billion parameters, which managed 76.2 percent. Fifteen times smaller, slightly better, and delivered three and a half months sooner. What required a data center in spring now fits on a workstation.

The pressure is coming from both sides. And that’s new. Open-model providers have moved fast: Moonshot released Kimi K3, the largest open model to date, while DeepSeek and Zhipu have placed their top models under the MIT license. Alibaba is going further by opening its highest-tier model class for the first time with Qwen3.8-Max. Meanwhile, closed providers are slashing prices and publishing stances on open weights – moves they didn’t see as necessary a year ago. From an operator’s perspective, it’s a rare alignment: both camps are chasing the same curve.

The forecasts point the same way. Gartner expects inference on a trillion-parameter model in 2030 to cost over 90 percent less than in 2025, driven by more efficient semiconductors and infrastructure, better model design, higher chip utilization, specialized inference hardware, and the use of edge devices for specific use cases. Compared to similarly sized models from 2022, analysts anticipate up to a hundredfold gain in cost efficiency. The outlook for local processing is similarly stark: by 2029, over two-thirds of companies are expected to run AI at the edge, up from roughly ten percent in 2025.

There is a caveat, though. Gartner flags it itself: the savings won’t flow one-to-one to the end user. Agentic applications consume far more tokens per task than classic text generation, clawing back a significant slice of the price advantage. Cheaper per token doesn’t automatically mean cheaper per process.

For planning purposes, the takeaway is still clear. The table below describes a snapshot rather than a steady state: its rows will slide downward, in months rather than years. Anyone locking in an architecture today should plan for local inference to grow as a share of the process landscape. Treating it as a niche for special cases is too narrow a view.

Where to Run What

What isn’t specified in any spec sheet can still derail projects. Power draw is the most common silent blocker: a rack that trips the floor’s circuit breaker isn’t an operational concept. Heat and noise become hard architectural stoppers when there’s no server room. And model updates can retroactively push requirements higher – documented in one case by an extra 10 GB of video memory compared to the previous version.

Open Weights Are Not Open Source

Open Weights means the weights are downloadable. What is permitted under this license is specified in the terms. Training data, training code, and reproducibility are not included.

Open Source under the OSI definition (OSAID 1.0) additionally requires Data Information: origin, scope, procurement and selection methods, labeling, processing, and filtering. By this standard, practically no current top-tier model qualifies as Open Source.

The differentiating factor: GLM-5.2 and DeepSeek V4 Pro are licensed under MIT, a genuine OSI-compliant license. Yet, they do not meet the OSAID criteria for Open Source AI because the training data is missing. A permissive license on the weights and truly open-source AI are two distinct concepts.

The EU AI Act addresses this with nuance. Recital 104 requires publicly available parameters, weights, architecture, and usage information for the exception. However, the exception only covers transparency-related obligations. Summarizing training content and compliance with copyright law remain mandatory.

In one sentence for decision-makers: Open Weights do not eliminate lock-in. They merely shift it from the model provider to your own operations – including patching, quantization, monitoring, and personnel.

What Really Drives the Decision

It’s not the number of employees. A tax advisor with twelve staff members handles client data that must never leave the premises. A corporation with a zero-trust cloud standard remains hybrid, even with ten thousand employees. The threshold lies elsewhere:

1. Data classification. Public and non-critical data can be sent to an API. Personal, business-critical, or confidential data pulls the entire stack in-house.
2. Load and predictability. Sporadic usage follows a different cost model than sustained demand. The break-even point for self-hosting versus top-tier API pricing is roughly 2 to 5 million tokens per day over twelve months. Against low-cost open-model providers, the threshold rises to 50 million tokens per day or more.
3. Model class versus task. Does a 27-billion-parameter model with RAG over internal documents suffice? Or does the task truly demand top-tier quality? Many business applications stop at “workable,” not “data-center grade.”
4. Operational maturity. Running your own inference stack costs 10 to 20 hours of maintenance per month. Without a team that supports it around the clock, a local pilot has an expiration date.
5. Latency and offline requirements. Manufacturing, field service, and air-gapped networks demand locality regardless of token price.
6. Power, cooling, and space. Raw GPU costs account for only 30 to 40 percent of the true investment. A rule of thumb is a factor of 2.5 to 3 on top of hardware pricing. Real-world production utilization hovers between 40 and 65 percent. Slide decks still tout 80 to 90 percent.
7. Regulation and contractual obligations. Data-processing agreements, the AI Act, industry rules, and customer contracts impose limits that no benchmark can override.

Hybrid is the norm: sensitive workloads run locally or in a private cloud, while general tasks leverage APIs – routed by data class rather than news cycle.

Key Takeaways

The debate is clearly divided. On one side, advocates rightly point out that open models enable verifiability and allow operations on your own hardware without surrendering data or intellectual property to closed providers. From the venture capital community comes the assessment that top-tier models are overkill for the vast majority of use cases. The opposing view argues that many of these models simply won’t run locally due to resource constraints. Both sides are correct – they’re just talking about different model classes.

For your own infrastructure, this leads to three steps:

First: Inventory use cases by data class and load. Which requests can leave the building, and which require offline capability or fixed latency? Without this list, any hardware discussion is pure speculation.

Second: Match the model class to the task rather than to the headline. For internal search, summarization, document Q&A, and coding assistance with RAG using your own documents, a model in the 24–32 billion parameter range on a single GPU is often sufficient. The trillion-parameter class remains API territory in practice unless someone budgets and operates a cluster with eight or more GPUs per node.

Third: Run the numbers with honest factors. Hardware cost multiplied by 2.5 to 3, utilization between 40 and 65 percent, 10 to 20 operating hours per month, break-even against the actual price tier used. Comparing GPU list prices directly to token prices produces a calculation that won’t hold up in real-world operations.

And fourth – with the longest-lasting impact: design the architecture for mobility. If a model that fits on a single graphics card outperforms its 15× larger predecessor within a quarter, locking into pure API operation is a bet against your own cost curve. This doesn’t mean bringing everything in-house today. It means building abstraction layers that allow switching: replaceable inference endpoints instead of hardwired provider SDKs, your own evaluation cases instead of third-party benchmarks, and a data classification system that already knows which processes will move inward first.

The community will keep evaluating models before weights are available – that’s noise. The signal beneath is speed: the gap between top-tier and locally runnable models is shrinking by the month. Downloadability is a licensing question; operability is an infrastructure question. But the answer to the second is shifting faster than most procurement cycles.

Frequently Asked Questions

Is a 27-billion-parameter model sufficient for enterprise tasks?

For internal search, summarization, document Q&A, and coding assistance with RAG over your own documents, this class often suffices. It runs on a single GPU with 16 to 32 GB of memory. What matters most is measuring against your own cases: what ranks well in a benchmark may not lead in your specific domain.

Why do hardware plans so often fail because of Mixture-of-Experts?

The number of active parameters determines compute cost per token, but memory demand follows total size. A model with 13 billion active parameters can load 284 billion into RAM. Basing procurement on the first figure can land you an order of magnitude off.

At what volume does self-hosting become cost-effective compared to an API?

Against top-tier model prices, break-even is roughly 2 to 5 million tokens per day over twelve months. Against low-cost open-model providers, the threshold rises to 50 million or more, since they operate the same hardware with better utilization and thinner margins. Both figures assume realistic utilization rates.

What does the development pace mean for ongoing procurement decisions?

The locally runnable model class is catching up to the top tier in months. Locking into pure API operation with multi-year commitments and hardwired provider interfaces weakens your negotiating position. A better approach is an architecture with replaceable inference endpoints that keeps the door open for change.

Image source: AI-generated (August 2026)