llama.cpp MTP Support: Local 27B Models 1.7x Faster on Consumer GPUs

Pull request 22673 introduces Multi-Token Prediction into llama.cpp’s main branch.

By Alec Chizhik May 27, 2026 7 min read

llama.cpp MTP Support: Local 27B Models 1.7x Faster on Consumer GPUs

7 min read

On May 16, 2026, PR #22673 was merged into llama.cpp. Multi-Token Prediction has been running in the mainline ever since. On an RTX 3090, Qwen3.6 27B performance jumps from 38 to 65 tokens per second, a solid 1.7x speedup. This isn’t hyperscaler news, but it shifts a calculation that many DACH teams have been weighing for the past year and a half: when does local inference pay off, and when does it not?

Key Takeaways

MTP is not a magic bullet for the entire model universe. The speedup only applies to models trained with MTP heads, currently Qwen3.5, Qwen3.6, DeepSeek V3, DeepSeek R1, and Gemma 4.
The 1.7x figure is real, but hardware-dependent. It originates from a RunPod RTX 3090 setup with Qwen3.6 27B. On other cards and with different quantizations, the value ranges between 1.4x and 2x.
The threshold for the token-cost debate is shifting. Local inference with an in-house consumer GPU is becoming cheaper, without H100-class hardware suddenly becoming obsolete. Where the switch makes sense and where it doesn’t.

What MTP does technically and why it must be built-in

Multi-Token Prediction is essentially a form of speculative decoding where a second prediction network doesn’t run externally, but sits directly in the main model as an additional output head. The MTP head proposes multiple tokens simultaneously; the main model verifies these in the next forward pass, accepting or discarding them. If the hit rate is high, the number of expensive forward passes per generated word drops measurably.

The decisive difference compared to classic speculative decoding with a separate draft model is highly practical. You don’t need a second model that you have to load in parallel and keep in VRAM. For 27B models on a 24 GB card, this saves exactly the headroom that the draft model would otherwise consume. In practice, this means an RTX 3090 or 4090 gains a speedup with MTP that it could never have achieved cleanly with a separate draft model due to VRAM constraints.

The flag in llama.cpp has been –spec-type draft-mtp since the merge. The patch comes from am17an, and the conversation surrounding PR #22673 clearly shows that while the code is functional, the MTP head must be present in the model checkpoint. Without an MTP head, the flag simply does not take effect.

What the 1.7x figure means in practice

The much-cited 1.7x mark comes from a RunPod benchmark using Qwen3.6 27B on an RTX 3090. Without MTP, the setup delivers 38 tokens per second in a single-stream. With the MTP head enabled, that figure hits 65. That is a difference of 27 tokens per second. It makes the difference between an interaction that feels fluid locally and one that drags along like a typewriter.

38 → 65 t/s

Qwen3.6 27B single-stream on RunPod RTX 3090, 1.71x throughput jump by activating the MTP head.

Source: Community benchmark, documented on Hugging Face and in llama.cpp PR #22673.

Anyone looking to extrapolate this to other hardware should be cautious. On a 4090, the curve shifts because the forward pass is faster there anyway, making the relative gains from MTP appear smaller. Initial community reports there suggest a factor of 1.4x to 1.6x. For Gemma 4 26B, reports indicate a speedup of around 40 percent. A realistic corridor for the coming weeks is 1.4x to just under 2x, depending on the model, card, and quantization level.

Which models support MTP today and which do not

MTP is not a switch you can simply flip on any GGUF file. The head must have been integrated during pre-training or fine-tuning; otherwise, there is simply no prediction for the main model to consume. As of today, this covers a very manageable list of models.

MTP natively available

Qwen3.5 from 7B upwards
Qwen3.6 27B and A3B-MoE variants
DeepSeek V3 and R1
Gemma 4 26B in the A4B variant

No MTP head, no speedup

Llama 3 in all sizes
Mistral and Mixtral families
Older Gemma versions prior to Gemma 4
Custom fine-tunes without an explicitly trained head

If you have built your setup on Llama-3-70B over the last few months, you won’t gain anything from the merge. The question is not whether to enable MTP, but whether you are prepared to switch your base model. For code generation and tool-calling, there is currently a strong case for looking at Qwen3.6. For classic RAG pipelines that have been running stably on Llama 3 for a year, a switch would be a purely cost-saving exercise that introduces other risks, ranging from prompt behavior to tokenizer differences.

Hardware reality between 3090, 4090, 5090 and H100

The intriguing question is not whether MTP still shows an effect on an H100. The answer there is predictably less spectacular. The real question is where the threshold lies at which a consumer GPU plus MTP can seriously be considered an inference replacement for an API hour.

On an RTX 3090 with 24 GB VRAM, Qwen3.6 27B in Q4 quantization with MTP reaches around 65 tokens per second, with context lengths stable up to 8k. That is sufficient for team-level code assistance, medium-length RAG responses, and structured extraction. What it is not enough for is simultaneous multi-user operation with high concurrency. As soon as three employees send requests in parallel, the experienced speedup drops again.

The 4090 only partially solves this. More bandwidth, more computing power, but the same 24 GB class. Anyone seriously looking to serve multiple streams in parallel will end up with the 5090 with 32 GB or directly with H100/H200. And this is exactly where many teams underestimate the situation: MTP is an answer for the single-stream case, not for high-load multi-client operation. In the data center, the H100 remains the more honest solution. On the engineer workstation or in a small in-house cluster, the 4090 with MTP suddenly becomes more interesting than it was three weeks ago.

When local inference really pays off against hyperscalers

The obvious calculation seems simple at first. A 4090 costs around 1,700 Euro to purchase in-house. Add electricity, maintenance, and depreciation over three years. If the setup handles several million tokens per day, it is cheaper than any API hour from OpenAI, Anthropic, or AWS Bedrock. That is what the slide says. But it is not entirely honest.

What the slides rarely show is the operational reality. Local inference is never just the card. It is also the team that updates the model. It is the responsibility for quantization updates, for the question of whether Q4_K_M or Q8_0 is the right level for the next quarter’s needs. It is the logging pipeline that needs to be made monitoring-ready. It is the willingness of someone to maintain the inference stack instead of just using it.

I have operated local inference seriously too early once before. That was in 2024, with a 13B model on a 3090. The cost comparison on paper looked good, but in reality, three engineering hours per week of maintenance ate up the cost advantage. Today, with MTP and models that do their job more reliably at 27B, the calculation is changing. But it does not change on its own.

In my view, the honest corridor looks like this: Local inference with an MTP-capable 27B model is worthwhile if the team can invest at least one quarter-day per month of engineering time into the stack, if workloads are concentrated on a few streams per hour, and if the responses do not have to compete in every second of absolute peak demand. It is not worthwhile if the response quality of GPT-5 or Claude 4 is expected, if multi-tenant concurrency is required, and if the team is not prepared to view GGUFs as an ongoing task.

Frequently Asked Questions

What exactly is Multi-Token Prediction in llama.cpp?

MTP is a form of speculative decoding where an additional output head integrated directly into the main model proposes multiple tokens simultaneously. The main model verifies these during the next forward pass, either accepting or discarding them. Unlike classic speculative decoding, no second draft model needs to be loaded in parallel.

Which models can I accelerate with MTP today?

Models that received an MTP head during pretraining. Specifically, these currently include Qwen3.5 from 7B upwards, Qwen3.6 including 27B and A3B-MoE, DeepSeek V3, DeepSeek R1, and Gemma 4 26B-A4B. Llama 3, Mistral, and older Gemma versions do not have an MTP head and will not benefit.

How significant is the speedup really?

The frequently cited 1.7x figure comes from a Qwen3.6-27B benchmark on an RTX 3090, moving from 38 to 65 tokens per second. On other cards and with different models, community reports suggest a range between 1.4x and 2x. Gemma 4 26B tends to hover closer to 1.4x.

Which flag in llama.cpp activates MTP?

Since the merge of PR #22673 on May 16, 2026, the flag is called –spec-type draft-mtp. The prerequisite is that the loaded model actually contains an MTP head in its checkpoint. Without the head, the flag will not take effect.

Is MTP enough to replace hyperscaler APIs?

For single-stream workloads on an engineering workstation or a small in-house cluster, MTP significantly shifts the cost-benefit analysis. For multi-tenant high-load environments or absolute peak quality, API models remain the more pragmatic choice for now. The operational reality-maintaining GGUFs, quantization tuning, and monitoring-continues to consume valuable engineering time.

Editor’s Reading Recommendations