llm-compressor | Heidelberg AI Curriculum

Development & infrastructurevLLM project contributorsOpen source

llm-compressor app decision matrix
Overview	LLM Compressor is a free, open-source library from the vLLM project that quantizes models (GPTQ, AWQ, SmoothQuant and similar algorithms) to cut memory use by roughly 50-75% and speed up serving. Pick it once you already have a working model on vLLM and need it smaller or faster. Note: its once-documented sparsity support (2:4 sparse compression) has been dropped for lack of hardware support and interest.
Other good fits	Teams already committed to vLLM who want a documented, repeatable quantization recipe instead of an ad hoc conversion.
Pros	Read pros LLM Compressor is an open-source library, maintained by the vLLM project, for optimizing models before deployment. Its main job is quantization: reducing numeric precision with algorithms like GPTQ, AWQ and SmoothQuant to shrink a model's memory footprint (the project states roughly 50-75% reduction) and speed up inference. It's built specifically as a companion to vLLM serving, not a general-purpose compression tool for any runtime. Pick it when… Your model works but is too large or slow to serve efficiently on the hardware you have. You're already using vLLM and want a documented quantization recipe rather than a one-off script. You need to compare a compressed model's answers against the original before deciding it's good enough.
Cons	Read cons Compressing a model changes it: quality, compatibility and operational behaviour can all shift, and a recipe that worked well for one model doesn't guarantee the same result on a different model, prompt distribution, precision or serving setup. The project has also narrowed its scope: sparse compression (including 2:4 sparsity) is no longer supported, due to limited hardware support and user interest, so don't plan around that feature if you saw it mentioned elsewhere. There's no published minimum GPU, CPU or memory requirement, or a guaranteed vLLM compatibility matrix; treat both as things to verify for your own setup. Pick something else when… You just need to serve a model that already fits your hardware. vLLM alone, without compression, may be all you need. You want a local runtime with its own quantized formats built in. llama.cpp has its own quantization approach (GGUF) suited to CPU and consumer GPU use. The conversion and validation work isn't worth it for your case. A smaller model that's already released at the size you need skips the compression step entirely. Avoid if You need sparse (2:4) compression; that support has been removed from the project.
Workflow	Select a supported model and your target runtime → choose a documented recipe → compress a copy of the model (never your only copy) → load it with vLLM → compare a set of test prompts against the original model's answers on the same prompts. A human still decides whether any quality loss is acceptable for the use case.
Cost	The library itself is free and source-available through its public repository. What you pay for is compute time to run the compression process, storage for both the original and compressed model artifacts, and evaluation time to check quality afterward. The project doesn't publish a fixed cost or benchmark for any specific recipe; it depends on your model size and hardware.
Restrictions	No minimum GPU, CPU or memory requirement is published, and there's no guaranteed compatibility matrix against every vLLM version; test your specific model, recipe and vLLM version together before relying on the result.
First project	Documentation-based suggestion — not run here. Compress a small model and check what changed. Take a small, publicly available model you can compare before and after. Apply one documented quantization recipe to a copy of it. Write five test prompts that represent your actual use case. Run all five through both the original and the compressed model, and compare the answers side by side. Done when you have a written comparison of all five prompts, not just an assumption that the compressed model is "probably fine" because it loaded successfully.
Links & evidence	LLM Compressor documentation (opens in a new tab)Why use LLM Compressor? (opens in a new tab)LLM Compressor repository (opens in a new tab) Documentation-based · not hands-on · Sep 23, 2026 Rests on LLM Compressor's documentation and GitHub repository, checked 23 September 2026. No model was compressed and no comparison was run for this review. The documentation's previously cited getting-started/quickstart URL no longer resolves; this review links the current documentation pages instead. LLM Compressor documentation · Sep 23, 2026 — Quantization algorithms (GPTQ, AWQ, SmoothQuant), vLLM companion role, dropped sparsity support. Why use LLM Compressor? · Sep 23, 2026 — Rationale and memory-reduction figures (roughly 50-75%). LLM Compressor repository · Sep 23, 2026 — Source-available project maintained by the vLLM community. Further review notes & alternatives Alternatives and trade-offs vLLM wins on its own when the uncompressed model already fits your hardware and serving needs. llama.cpp wins when you want a local runtime with its own built-in quantization formats, especially for CPU or consumer-GPU use. A smaller pre-released model wins when the compression and validation effort isn't worth it for your project's scale. Learn more LLM Compressor documentation (opens in a new tab) Why use LLM Compressor? (opens in a new tab) LLM Compressor repository (opens in a new tab) Catalogue facts Compress Hugging Face checkpoints for vLLM deployment. Apply quantization, pruning, and KV-cache techniques. Save models in the compressed-tensors format for inference. vLLM project contributors · free · self-host quantization pruning model compression vllm compressed-tensors open-source View sourced image llm-compressor Press Escape to close. At 6:20 in Red Hat’s official LLM Compressor demonstration, OpenShift AI shows a succeeded compression/evaluation pipeline and its model artifact. OpenShift AI is the documented companion execution interface, not a GUI bundled with the Python library; this is the publisher’s recorded run. Red Hat Developer — LLM Compressor in OpenShift AI demonstration Owner-requested editorial use of an official public demonstration frame, 19 September 2026. Copyright remains with Red Hat/the original publisher; attribution preserved. Complete video frame extracted at 380 seconds without overlays. No local compression run or independently verified benchmark is asserted. Converted on 15 October 2026 to a 1600px-wide WebP (quality 80) for faster page loads; the original image remains in the media library. Original image source (opens in a new tab)

Browse all tools