runtime · vLLM project — originated at UC Berkeley Sky Computing Lab (Kwon et al., PagedAttention, SOSP 2023), now a broad open-source community (2,000+ contributors)
vLLM
vLLM is the engine you reach for when 'a model on my laptop' has to become 'a model the whole lab hits at the same time.' Built at UC Berkeley's Sky Computing Lab and now maintained by 2,000+ contributors, its core trick is PagedAttention — managing the GPU memory used for each conversation's attention cache the way an OS pages virtual memory, so many concurrent requests share a GPU without fragmenting it — combined with continuous batching and prefix caching. The result is dramatically higher throughput per GPU than a naive single-request server under real concurrent load. It speaks an OpenAI-compatible API (plus an Anthropic Messages API and gRPC), runs 200+ Hugging Face model architectures, and supports FP8/INT8/INT4/GPTQ/AWQ quantization to fit bigger models in less VRAM. For a lab: one GPU serving ten people's simultaneous questions instead of one.
What it can help with
- model serving
- gpu batching
- paged attention
- continuous batching
- openai api compatible
- quantization support
- multi-user inference
- hardware plugins