llm-d adds a router and cache-aware endpoint picking above model servers like vLLM, distributed across a Kubernetes InferencePool. Pick it once you already run multiple model-server replicas and have a real routing or caching problem to solve. The quickstart alone deploys 8 GPU replicas of a 32B model, so this is a cluster-operator tool, not a first local project.
Other good fits
Operators who already have a measured queueing, caching or scaling problem across multiple inference pods.
+ Pros
Read pros
llm-d sits above inference engines like vLLM or SGLang rather than replacing them. It defines a router, an InferencePool that groups model-server pods, and endpoint-picking logic that can use load and cache information to decide which pod serves a given request. It also documents autoscaling and batch-serving paths for distributed workloads on Kubernetes.
Pick it when…
You're running several model-server replicas on Kubernetes and need requests routed intelligently between them, not round-robin.
You have a measured caching or queueing problem across replicas that a smarter router could actually solve.
You already own Kubernetes operations and accelerator capacity as part of your job.
− Cons
Read cons
The routing layer is another operational component to run, upgrade and keep compatible with your Kubernetes and Gateway API versions. Cache-aware routing only helps where your workload and server metrics actually support it, and advanced prefill/decode separation needs specific interconnects and model support that not every setup has. Published project benchmarks don't establish an improvement for your own workload; measure your own baseline first. One successful test completion doesn't establish resilience, fairness between users, or a sound scaling policy, and idle cluster capacity is still a cost even with no requests arriving.
Pick something else when…
A single model server already has enough capacity. A single vLLM instance is simpler and has nothing extra to coordinate.
You're doing personal or small-scale experiments.Ollama or llama.cpp fit that far better than a Kubernetes cluster.
You'd rather not own cluster operations at all. A managed inference provider removes that ownership, at the cost of usage-based billing and external processing.
Avoid if
You don't have a budgeted Kubernetes cluster and accelerator capacity, or ownership of operating one.
Requirements
The documented quickstart needs kubectl, Helm, a Kubernetes cluster (the guide's "Standalone Mode"), a HuggingFace token to pull models, and the Gateway API inference-extension resources. It deploys on NVIDIA GPUs by default, using vLLM as the model server; AMD, Intel, TPU and CPU guides exist separately in the repository. Hardware, driver and model-server compatibility are deployment-specific and weren't measured for this review.
First run
Documentation-based suggestion — not run here.
Read first run
Confirm routing works in a test pool before trusting it with real load.
In an operator-prepared test namespace, follow the documented router and model-server deployment steps.
Wait for the pods to become ready and get the router's proxy address.
Send one completion request through the router, using the exact model identifier the pool is actually serving.
Check which model-server pod handled it, using your cluster's existing observability.
Done when you've confirmed one request round-trips correctly through the router to an eligible pod, and you understand which component (router vs. model server) was responsible for what. That confirms connectivity, not a throughput or latency improvement.
Integration
llm-d's router exposes an in-cluster inference endpoint; it doesn't decide external exposure, authentication or network policy for you; that stays under your cluster operator's normal controls.
Check the result
Connectivity is not a performance result: compare the pod's own logs and your cluster's latency dashboard against a baseline before claiming any improvement. If the pool isn't ready, resolve deployment status with the operator rather than provisioning more accelerator capacity just to get a sample answer.
Links & evidence
Documentation-based · not hands-on · Sep 23, 2026
Rests on llm-d's architecture documentation, quick start and project README, checked 23 September 2026. No cluster was provisioned and no request was sent for this review.
Project overview · Sep 23, 2026 — Distributed inference scope, cache/routing capabilities, distinction from underlying model engines.
Further review notes & alternatives
Costs and operating effort
llm-d itself is open-source and free. The real cost is everything around it: the quickstart's default manifest deploys eight GPU replicas of a 32B model, which is a meaningful capacity and cost commitment, not a light tutorial. Add cluster nodes, accelerators, storage, networking, model downloads and ongoing operator time. No official cost estimate or benchmark is published; budget based on your own replica count and hardware pricing.
Alternatives and trade-offs
vLLM wins on its own when a single instance or a small fixed set of replicas already covers your load.
Ollama wins for personal experiments and small local deployments where Kubernetes is overkill.
llama.cpp wins when you want a lightweight local runtime rather than cluster-scale orchestration.