Development & infrastructurevLLM project — originated at UC Berkeley Sky Computing Lab (Kwon et al., PagedAttention, SOSP 2023), now a broad open-source community (2,000+ contributors)Open source

vLLM

Decision matrix

vLLM runtime decision matrix
Overview

vLLM is a free, open-source engine that turns an open-weight language model into a private API on your own GPU server. It speaks the OpenAI and Anthropic API formats and serves many people and tools at once. Pick it for a shared team server on Linux; for one laptop, Ollama or LM Studio are much simpler.

Other good fits
  • Batch jobs, such as classifying thousands of documents overnight, run far faster than one request at a time.
Pros
Read pros
  • Pick it when several people or tools share one GPU server. vLLM batches their requests together, so the GPU serves many users at once instead of making them queue.
  • Pick it when your apps already speak the OpenAI API. vllm serve exposes /v1/chat/completions, /v1/completions and embeddings, plus an Anthropic-style /v1/messages route, so tools built for those APIs can use your own model.
  • Pick it for batch work. The Python LLM class runs a whole list of prompts in one go, for example 5,000 support tickets to classify overnight.
  • Pick it when you want a model's official serving recipe. Many model cards, such as Ornith-1.5-35B-A3B, give a vllm serve command first.
Cons
  • Pick something else when you're one person on a laptop: Ollama or LM Studio install in minutes on Windows and Mac.
  • Pick something else when the server must run Windows natively: vLLM supports Linux (WSL works). llama.cpp runs on Windows directly.
  • Pick something else when you serve across a Kubernetes cluster with many GPUs: llm-d adds cache-aware routing across pools of model servers.
  • Pick something else when several models must take turns on one small GPU: llama-swap starts and stops them on demand.

Avoid if

  • Nobody on the team can look after a Linux server, GPU drivers and upgrades every few weeks.
Requirements

The stable quick start lists Linux and Python 3.10-3.13, with separate NVIDIA, AMD, Intel, TPU, Ascend and Apple Silicon routes. Prepare the documented backend, model access, accelerator memory and disk for the exact release; no universal hardware minimum, throughput result or model compatibility claim is made here.

First run

Documentation-based suggestion — not run here.

Read first run

Give a 10-person lab one private chat model on a spare GPU workstation.

  1. Use Linux (or WSL on Windows) with an NVIDIA GPU of compute capability 7.5 or newer: a T4, an RTX 20-series card or later. In a Python 3.12 environment, run uv pip install vllm --torch-backend=auto.
  2. Start a small model to check the setup: vllm serve microsoft/Phi-4-mini-instruct --api-key <long-random-string>
  3. Test it: curl http://localhost:8000/v1/models -H "Authorization: Bearer <long-random-string>"
  4. Connect Open WebUI to http://<server>:8000/v1 with the same key, so colleagues get a chat page in the browser.
  5. When that works, switch to a bigger model that fits your GPU memory.
Integration

Use LLM and SamplingParams for local batches, or vllm serve <model> for an HTTP server. The OpenAI-compatible chat route requires a text-generation model with a chat template; interface compatibility does not make all parameters or models interchangeable. Keep local servers private until deliberate remote-access controls are in place.

Check the result

Expect one limited completion printed for the sample sign. Compare it with the prompt and inspect the selected backend/model configuration if initialization fails. This is an expected documentation outcome, not an observed inference result.

Links & evidence
Documentation-based · not hands-on · Sep 23, 2026

Based on the vLLM quickstart, GPU installation guide, OpenAI-compatible server reference and GitHub release list, checked 23 September 2026. We did not run it for this review.

  • vLLM quickstart · Sep 23, 2026 — Install with uv, offline batch with the LLM class, vllm serve for an OpenAI-compatible server.
  • vLLM GPU installation · Sep 23, 2026 — Linux only (WSL on Windows), Python 3.10-3.13, NVIDIA compute capability 7.5 or higher; AMD, Intel and Apple Silicon routes.
  • vLLM OpenAI-compatible server · Sep 23, 2026 — Chat, completions and embeddings routes; Anthropic messages API at /v1/messages.
  • vLLM releases · Sep 23, 2026 — v0.30.0 on 22 September 2026, v0.29.0 on 9 September, v0.28.0 on 26 August: about two weeks apart.
  • TGI maintenance notice · Sep 23, 2026 — Hugging Face recommends vLLM as a successor to TGI.
Further review notes & alternatives

Costs and operating effort

Prices checked 23 September 2026: free software (Apache-2.0); you pay for the machine. You need a Linux machine with a GPU that holds the model plus working memory. Rule of thumb: in 16-bit, weights take about 2 GB per billion parameters, so a 4B model wants a 16 GB card and a 30B model two 40-80 GB cards unless you use a quantised version. Rented cloud GPUs are billed by the hour.

  • A new release arrives about every two weeks (v0.30.0 on 22 September 2026). Flags sometimes change, so pin the version and test before you upgrade.
  • You look after the server: drivers, CUDA, disk for model downloads (tens of GB per model), monitoring through the built-in Prometheus metrics, and access.
  • --api-key is one shared key, not user accounts. Keep the server behind a firewall or VPN.

Alternatives and trade-offs

  • Ollama: wins for one machine and quick model downloads; weaker when many people use it at once.
  • llama.cpp: wins on CPUs, Macs and mixed hardware with GGUF files.
  • Text Generation Inference: only if you already run it. Hugging Face has put it in maintenance mode and recommends vLLM.

Learn more

Catalogue facts

Serve language models to many simultaneous users from shared GPU hardware. Use paged attention, batching, caching, and quantization to manage inference. Expose OpenAI-compatible, Anthropic, or gRPC interfaces for applications.

vLLM project — originated at UC Berkeley Sky Computing Lab (Kwon et al., PagedAttention, SOSP 2023), now a broad open-source community (2,000+ contributors) · free · self-host

  • model serving
  • gpu batching
  • paged attention
  • continuous batching
  • openai api compatible
  • quantization support
  • multi-user inference
  • hardware plugins
  • automate
  • data-analytics
  • open-source

Browse all tools