runtime
llm-d
llm‑d is an open‑source, CNCF‑sandbox project that lets you serve large language models at scale on any accelerator (GPU, TPU, XPU, CPU, etc.) using Kubernetes. It provides ready‑made Helm charts and “well‑lit paths” that integrate model servers such as vLLM and SGLang, turning single‑node engines into production‑grade serving clusters. The software is free under the Apache 2.0 license and runs wherever you can run a Kubernetes cluster.
What it can help with
- distributed inference
- kubernetes
- helm charts
- scale out
- vllm