app · vLLM project contributors
llm-compressor
LLM Compressor is a Python library that applies state‑of‑the‑art compression algorithms (quantization, pruning, KV‑cache tricks) to large language models so they run with lower memory and faster inference on GPUs via vLLM. It is installed with pip install llmcompressor and works with Hugging Face model checkpoints, saving compressed models in the compressed‑tensors format. The tool is free and open‑source under Apache‑2.0.
What it can help with
- quantization
- pruning
- model compression
- vllm
- compressed-tensors