Heidelberg AICurriculum

Llama.cpp Just Merged MTP And You Should Be Using It.

Tim Carambat ·2026-05-18 ·2 min read

Summary written by us from the video's transcript. The video, and everything in it, is Tim Carambat's work.

Learn how to enable and use Multi‑Token Prediction (MTP) in llama.cpp for faster local inference without accuracy loss.

Takeaways

  • MTP adds multi‑token ahead prediction to a single model, delivering ~25% faster token generation without accuracy loss.
  • Only the newest llama.cpp build can load MTP‑enabled GGUF models; older builds will crash.
  • Download the explicit “MTP” version of a model (e.g., 9B‑MTP.gguf) from Hugging Face to use the feature.
  • Enable MTP with `--spec-draft` and set ahead tokens via `--spec-draft-n-max`; start with 1 token for best speed/accuracy balance.
  • Higher `n-max` values can hurt performance on limited hardware—benchmark your setup before increasing.

What is MTP and why it matters

Multi‑Token Prediction (MTP) lets a single model generate several tokens ahead, boosting throughput by roughly 25% or more with no trade‑offs. It builds on speculative decoding, which previously required a small “draft” model plus the main model, but MTP removes the need for a separate draft model.

Because it works with any hardware—though better hardware yields larger gains—MTP expands the practical speed of locally run models such as those in llama.cpp.

Supported models and formats

MTP is already baked into several GGUF/ggml releases: DeepSeek V3 and V4, Nematron 3‑Super/Ultra, Qwen 3.5 and 3.6 series, and the Gemma 4 line (though Google’s published GGUF files currently lack the flag).

MOE models (e.g., a 122‑active‑10B mixture‑of‑experts) may see little or no speed benefit; dense models like Qwen are recommended for testing MTP.

Getting the right software and model files

You need the latest llama.cpp build (merged MTP support two days ago). Older builds will refuse to load an MTP‑enabled GGUF, producing a crash.

Download the specific “MTP” version of a model from Hugging Face—e.g., the *9B‑MTP.gguf* file for Qwen 3.5—instead of the regular GGUF. The non‑MTP files will still run but cannot use the new speed feature.

Running llama.cpp with MTP

Launch the server (or CLI) with two new flags: `--spec-draft` to enable MTP and `--spec-draft-n-max <N>` to set how many tokens ahead to predict.

Typical values are 1‑token ahead (gives ~25% speedup), 3‑tokens ahead (~8% boost), while higher numbers like 6 can actually slow you down on modest hardware.

Tuning and benchmarking

In a test with Qwen 3.5 9B on an M4 Pro (48 GB RAM), baseline throughput was ~45 TPS. Enabling one‑token MTP raised it to ~55 TPS (+25%). Three tokens gave ~49 TPS (+8%), and six tokens dropped to ~30 TPS, illustrating diminishing returns.

Choose the `n-max` value that matches your hardware and workload; start with 1 and increase only if you see consistent gains.

Practical considerations

MTP works with vision models and parallel decoding, though prompt‑processing may still need optimization.

The feature is backward compatible: older llama.cpp versions can run non‑MTP models, but cannot load MTP‑enabled files. Keep both llama.cpp and model files up to date for best results.