▸How-to Everyone
Using Llama.cpp with the Arc GPU cuts prompt-generation time in half, raising token throughput from ~1,000 to ~2,200 t/s
Track 12 · Advanced
12.10
Quantization as an engineering decision
Somebody already chose your model's precision. It should be you.
1Overview
Storing a model's weights — and sometimes its activations — at lower precision, trading a measurable amount of quality for a large amount of memory.
When you download a model in a desktop runner, a quantization was picked for you and never mentioned. It is the single largest lever between "this fits on my card" and "this does not", and it is invisible in every other chapter of this course. → What the bit-widths actually do to weights, how to measure the quality you lost rather than trusting a rule of thumb, and how to produce a quantized model yourself. → The best-known tool in this space is archived; its successor is the anchor here, which is itself the lesson.
The lever nobody shows you. Bit-width decides whether a model fits your card at all, and a desktop runner picks it silently. Here you pick it deliberately, produce the artefact yourself, and measure what the compression cost you on your own task rather than on somebody's benchmark.
1.1Before you start
1.2After this chapter you can
→Explain what quantization changes about a model and what it does not
→Estimate memory from parameters, bit-width and context before downloading anything
→Produce a quantized model at a chosen precision
→Measure quality loss on your own task instead of on a public leaderboard
→Choose between weight-only and activation quantization for your hardware
→Tell an actively maintained quantization toolchain from an archived one
1.3When to reach for it
Whenever the model you want is larger than the memory you have, and whenever you are about to buy hardware to avoid that problem.
1.4Key parts
A source model, a quantization method, a calibration set, an output format your serving engine reads, and an evaluation you wrote yourself.
1.5Free vs paid
Entirely free and open. The cost is compute time for the conversion and an afternoon building the evaluation that makes the result meaningful.
1.6Watch out
Quality loss is task-dependent. A quantization that is invisible on chat can be obvious on structured output or code, and the public benchmark will not tell you which you have.
2Lessons 8
2.1 Explain how quantisation affects a model
An 8‑bit quantisation reduces each model weight to eight bits while keeping the parameter count unchanged.
State exactly what lower precision changes in a model’s weights and note that it never alters the parameter count
- Identify the current precision of the model weights (e.g., fp16 or bf16)
- Apply an 8‑bit quantisation to the model using a conversion tool
- Compare the resulting file size with the original to see the memory reduction
- You'll see The same open‑weight model appears in three files on Hugging Face – an fp16 original and two GGUF quantisations – with sizes roughly matching bits‑per‑weight
- Takeaway Quantisation reduces bits per weight without changing parameter count or architecture, lowering memory use and usually speeding inference at a quality trade‑off
- Check What specific change does applying an 8‑bit quantisation make to a model’s weights while leaving the parameter count unchanged?
- Cost Free to reason about — quantization is arithmetic on numbers you already have. The cost shows up later, as GPU-hours to run the conversion and as whatever quality the compression actually cost you.
2.2 Calculate total memory required for a quantised model
Weight memory is computed as parameters × bits ÷ 8, and KV‑cache size derives from sequence length, layers, heads and head dimension.
Determine whether a model’s weights and KV cache fit within your GPU memory before downloading.
- Compute weight memory using parameters × bits ÷ 8 and note the result in gigabytes.
- Estimate KV‑cache size by multiplying sequence length, number of layers, attention heads and head dimension, then converting to bytes.
- Add a safety margin for activations and overhead to the sum of weight and KV‑cache memory.
- Compare the total against your GPU’s available memory to decide if the chosen bit‑width and context length are feasible.
- You'll see A side‑by‑side comparison of weight memory at fp16, Q8 and Q4 alongside a growing KV‑cache estimate that eventually exceeds the low‑bit weight size.
- Takeaway Weight memory follows parameters × bits ÷ 8 while KV cache expands with context length, so both must be budgeted before download
- Check How do you calculate the total memory required for a quantised model, including both weight storage and KV‑cache expansion?
- Cost Free — this is arithmetic, not a run. Getting it wrong costs an out-of-memory error mid-session, or hardware bought a size larger than the actual workload needed.
2.3 Decode GGUF quantisation filenames
The GGUF filename parts—bit‑width prefix, optional _K k‑quant marker, and size suffix (_S, _M, _L)—describe the quantisation scheme.
Read a GGUF filename like Q4_K_M and know what each part means, so a desktop runner's silent default stops being unreadable.
- Identify the base bit‑width prefix (e.g., Q4, Q5, Q6) in the filename.
- Recognise the
_Kmarker that signals a k‑quant two‑level block scheme. - Determine the size variant suffix (_S, _M, _L) to infer the quality/size trade‑off.
- You'll see A model card offering
Q4_K_S,Q4_K_M,Q5_K_M,Q6_K, andQ8_0variants of the same model, with file sizes stepping up in that order and no accuracy number attached to any of them — which is itself the point of the next lessons. - Takeaway GGUF k‑quants (
Q4_K_M,Q6_K, …) pair a bit‑width with an improved two‑level block scheme and an S/M/L size variant; reading the name tells you the method, not the quality loss for your task - Check When you see a GGUF filename like
Q4_K_M, what does each segment (bit‑width prefix,_Kmarker, size suffix) indicate about the quantisation method? - Cost Free — this is reading comprehension for a filename you already download. No conversion is run in this lesson.
2.4 Compare GPTQ and AWQ quantisation approaches
GPTQ uses per‑layer Hessian reconstruction whereas AWQ scales activation‑salient channels before rounding.
Show how GPTQ and AWQ differ in optimisation strategy and calibration cost
- Clone the GitHub repository that implements the desired method
- Run the vllm-project/llm-compressor script with the
--method awqflag to quantise the model using AWQ - Run the ModelCloud/GPTQModel script with the
--method gptqflag to quantise the same model using GPTQ
- You'll see Two 4‑bit weight files for the same base model – one produced by GPTQ’s layer‑wise reconstruction and one by AWQ’s salient‑channel scaling
- Takeaway GPTQ optimises weights using a per‑layer Hessian while AWQ scales only activation‑salient channels before rounding, separating algorithm choice from tool implementation
- Check What is the core difference in optimisation strategy between GPTQ’s per‑layer Hessian reconstruction and AWQ’s activation‑salient channel scaling?
- Cost AWQ calibration is markedly cheaper than GPTQ's per the sources — quantifiable difference in wall-clock and calibration-set size, not free but a real gap between the two.
2.5 Quantise your own model
A source model, a target bit‑width or scheme, and (for GPTQ/AWQ) a small calibration dataset are needed to start the pipeline.
Run a full quantisation pipeline from source model to ready‑to‑use artefact using an up‑to‑date tool
- Select the appropriate tool (llama‑quantize, ModelCloud/GPTQModel or vllm‑project/llm‑compressor) based on your target runner and precision scheme
- Prepare the three required inputs – a full or half‑precision source model, the desired bit‑width or scheme, and for GPTQ/AWQ a small calibration dataset of general‑text samples
- Configure the tool: create a
GPTQConfig(or equivalent) with the chosen bits and group size, or build aQuantizationModifierrecipe specifying target layers and scheme - Execute the quantisation command – call
model.quantize(calibration_dataset, batch_size=1)for GPTQModel or runoneshot(model=model, recipe=recipe)for llm‑compressor - Save the resulting artefact with
model.save(quant_path)or the tool’s output routine and load it in a compatible inference engine
- You'll see The log shows the calibration data loading, a progress indicator for each layer (or a single pass), and finally a smaller model directory together with a config file describing the bit‑width and other settings
- Takeaway Quantisation always requires a source model, a target scheme and, for GPTQ or AWQ, a calibration set – the process is performed by configuring the tool, invoking its quantise command and saving the resulting artefact
- Check Which three essential inputs must you provide before running a full quantisation pipeline for your own model?
- Cost GPU time — the calibration pass over a few hundred to a few thousand samples, typically minutes to an hour on a single consumer GPU for a small-to-mid-size model, longer for GPTQ's heavier per-layer optimization than for AWQ's lighter scaling pass.
2.6 Compare model outputs before and after quantization for your own tasks
Side‑by‑side generation of original fp16 and quantised outputs on a representative prompt set enables task‑specific quality comparison.
Create a side‑by‑side evaluation of the original and quantized models using real prompts you actually use
- Gather twenty to fifty representative prompts that you use in production, keeping them exactly as they are written
- Run each prompt on the original fp16 model and capture the outputs
- Run the same prompts on each candidate quantised model and capture those outputs
- Score the results using your real‑world criteria (JSON parses, code runs, numbers match, answer is acceptable)
- Compare the scores to spot where the quantised model deviates from fp16
- You'll see The same set of prompts produces nearly identical chat responses, but shows parsing failures on structured‑output prompts when run with the quantized model
- Takeaway Leaderboard scores hide task‑specific loss, so a small custom eval built from your own real prompts reveals where quantisation truly hurts
- Check How can you set up a side‑by‑side evaluation to compare original fp16 outputs with those of a quantised model on your real‑world prompts?
- Cost An afternoon to build the eval set once, then minutes to re-run it against each new candidate. Cheaper than shipping a quantization that silently breaks the one task it needed to handle.
2.7 Decide between weight‑only and activation quantization based on GPU support
A GPU with compute capability ≥ 8.9 is required for full weight‑and‑activation (W8A8) execution; lower caps fall back to weight‑only.
Select the appropriate quantisation strategy that matches your hardware’s compute capability
- Check the compute capability of your GPU using the system information tool
- Load the FP8‑quantised model with vLLM on that GPU
- Observe whether the runtime reports full W8A8 execution or silently falls back to weight‑only
- You'll see The same FP8 model runs as full W8A8 on a GPU with compute capability 8.9 and falls back to weight‑only on a GPU with compute capability 7.5
- Takeaway Weight‑only quantisation works everywhere while activation quantisation needs compute capability ≥ 8.9 for its speed benefit
- Check What GPU compute capability threshold determines whether activation quantisation will run as full W8A8 instead of falling back to weight‑only execution?
- Cost No extra cost to check compute capability before you commit to an activation-quantized target — one lookup against the GPU's spec sheet. Getting it wrong costs the throughput you thought you bought.
2.8 Decide whether a quantisation library is safe to use
Recent commits, absence of an archived banner, and active issue discussion indicate a quantisation library is safe to use.
Identify actively maintained quantisation projects and avoid abandoned ones before building pipelines
- Open the library’s page on GitHub
- Verify there is no archived banner at the top of the repository page
- Inspect the Commit history tab for a commit within the last few months
- Search the Issues list for recent activity and look for maintainer replies to maintenance questions
- If an archive banner exists, read it for the recommended successor link
- You'll see A repository page on GitHub without an archived banner and with recent commits shown in the commit history
- Takeaway Treat a quantisation tool like any other dependency by checking for recent commits and the presence of an archive banner before relying on it
- Check What repository signals—such as recent commits, lack of an archived banner, and active issue replies—indicate that a quantisation library is safe to adopt?
- Cost One page load per dependency, done once before you build on it and again before you'd call it safe to keep depending on. Free. The cost of skipping it is a pipeline nobody can get a fix for when something breaks.
3You’ll know it worked 13 checkable outcomes in this chapter
- ✓Token generation rate increases to ~2,200 tokens per second when the Arc GPU is enabled
- ✓Token generation rate falls to ~1.4 t/s when the model is split
- ✓Token-generation throughput is within ~6%
- ✓Prefill units differ by ~1,350
- ✓Running a forward pass shows model outputs within a few percent of the original FP32 model
- ✓File size drops and inference runs without errors on the target device
- ✓Validation accuracy after QAT matches or exceeds post‑training quantized model; file size remains reduced
- ✓Check the checkpoint size; it should be ~1 GB smaller than an equivalent legacy‑quantized model of the same parameter count
13 outcomes in all — one per recipe below.
4FAQ, Tips & How-to 22
one problem, one solution, one action
▸How-to Everyone
Distributing a 75 GB Llama 3 70B model across three NUCs allows it to run, but token throughput drops to ~1.4 t/s due to network latency per token
▸How-to Everyone
Placing a full copy of a model on each node and load-balancing requests yields near-linear scaling: three nodes reach ~500 t/s versus 196 t/s on a single machine
!Tip Everyone
Token-generation speed — Llama CPP Gemma-412B on Halo vs Spark
Halo achieves near-equal token generation to the Spark
!Tip Everyone
Prefill speed gap — Spark ~2,000 units vs Halo ~650 units
Halo is about one-third as fast for compute-heavy prefill tasks
!Tip Everyone
GGUF file — a zip-like container for LLM weights and metadata
A GGUF file bundles everything needed to run an LLM, so downloading it is usually all that's required
▸How-to Everyone
You can estimate how much GPU memory the weights alone will consume
▸How-to Everyone
Context memory often exceeds weight memory; estimate it to avoid GPU OOM
▸How-to Everyone
Want multi‑token predictions from the server or CLI
Turn on multi-token prediction when running the server or CLI
▸How-to Everyone
Model predicts too many future words
Control how many future tokens the model predicts at once
▸How-to Everyone
Predicting one token ahead gives ~25% throughput boost on most hardware
▸How-to Everyone
Use a consistent test to decide the best n-max for your machine
▸How-to Everyone
Easily swap models by loading one .gguf file
▸How-to Everyone
Know that llama.cpp is the low-level engine powering all three runners
▸How-to Everyone
Model is large and slow on GPU
It maps the maximum absolute weight to the extreme integer range (‑128/127) and shifts all values so zeros stay zero, enabling a linear scaling of every weight into an 8‑bit integer. This reduces model size by ~4× and allows int8 hardware acceleration.
▸How-to Everyone
Model too large for fast inference
Symmetric quantization uses a single scale factor and zero point of zero to map floating‑point values in a symmetric range (e.g., –1 to 1) onto an integer range like –128…127. This preserves the sign distribution and simplifies dequantization, making it fast for inference on GPUs and CPUs.
▸How-to Everyone
Need to see how much accuracy drops after quantizing a model
Calibration runs a small representative dataset through the unquantized model to record activation ranges, enabling accurate per‑layer scales and zero points. By comparing outputs before and after quantization on this data you can quantify accuracy drop without full retraining.
▸How-to Everyone
Model won’t fit on a small device
Converts a trained TensorFlow SavedModel to a TFLite file with integer weights, reducing size dramatically (e.g., from ~300 KB to ~80 KB). It works by adding an optimizations flag before conversion.
▸How-to Everyone
Need a smaller model but don’t want accuracy loss
Wraps a regular model with `tf.keras.models.quantize_model` so that weights are simulated as low‑precision during training, then fine‑tunes briefly. This yields a quantized model whose accuracy stays close to the original.
▸How-to Everyone
Model still in old type‑zero format
Type‑zero uses symmetric linear quantization: it computes a single scale factor that maps the floating‑point range onto an integer range (e.g., -7…7 for 4‑bit). Each weight is stored as an int plus that shared scale, which lets you reconstruct approximate values at inference.
▸How-to Everyone
Checkpoint is bloated by per‑block scales
K‑quants introduce a two‑level scheme: regular 32‑weight blocks are grouped into an 8‑block “super‑block”. The per‑block INT4 weights keep their own scale, but those scales themselves are stored as INT8 and share a single FP16 super‑scale, cutting constant overhead roughly in half.
▸How-to Everyone
Model gets too inaccurate after quantization
An importance matrix scores each weight (or row) by how much its activation affects model output on a calibration set. During quantization you solve a small quadratic optimization to find adjusted scale S′ and zero‑point Z′ that minimize weighted MSE, giving higher fidelity where it matters most.
The same set on /recipes, filtered by tool and role.
5See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.