Hands-on: stand up your own gateway, then break it on purpose
An open-source LLM proxy: one OpenAI-compatible endpoint in front of many providers, adding virtual keys, budgets, routing, fallbacks and a spend log.
The gateway chapter argues the case. This one is the keyboard: install LiteLLM, put two providers behind one endpoint, cut a virtual key for a colleague with a hard budget, watch the 429 arrive when they spend it, and read the log that says who spent what. → It follows the same shape as the other per-tool chapters in this course — one product, start to finish, with the parts that bite you called out rather than smoothed over. → Note before you start: LiteLLM was rearchitected onto a Rust core with a Python SDK, so guides written before that describe a different deployment.
One product, end to end. Install it, route through it, cut a budgeted key, break it deliberately, and read what it recorded. By the end you have a gateway a small team can actually use — and you have seen each failure mode before it finds you in production.
Two or more AI projects, or two or more providers, and nobody can currently say who is spending what.
The proxy, a config file listing model routes, a database for keys and spend, virtual keys with budgets, and a fallback chain.
The proxy is open source and self-hostable at the cost of one container and a database. There is also a paid enterprise tier; nothing in this chapter needs it.
It sits on the critical path of every AI call your organisation makes. Anything that takes the gateway down takes all of them down, so it needs its own availability plan before production traffic does.
The litellm --config config.yaml command launches the LiteLLM proxy, exposing an OpenAI‑compatible endpoint that routes calls defined in a YAML file.
Start the LiteLLM gateway and receive a completion through it.
/chat/completions or /v1/chat/completionslitellm --config config.yaml command achieve when starting the LiteLLM gateway?model_list entries in config.yaml define each provider that the proxy can route to.
Add a second provider to the configuration and invoke both models using identical request shapes, varying only the model name.
model_list with the new provider’s detailsmodel to gpt-3.5-turbomodel field to claude-haikumodel_list without altering the caller‑facing APImodel_list let you invoke different providers using identical request shapes?A virtual API key created through /key/generate that includes a max_budget limit.
Generate a virtual key with a $0.01 spend limit, use it until the limit is reached, and observe the proxy’s budget‑exceeded error.
ExceededTokenBudgetlitellm_settings.fallbacks maps a primary model like gpt‑3.5‑turbo to fallback models such as claude‑haiku.
Create a fallback from one model to another, break the primary key, and confirm the request still returns a 200 response.
gpt-3.5-turbo to ["claude-haiku"] and set num_retries: 1gpt-3.5-turbo entry’s api_key line so it references a non‑existent environment variable, e.g. api_key: os.environ/THIS_VAR_IS_NOT_SETmodel: "gpt-3.5-turbo" using curl as you normally wouldmodel field reports claude-haiku and the HTTP status is 200model field shows claude-haiku instead of the broken gpt-3.5-turbofallbacks under litellm_settings redirects errors to alternate models without changing the caller’s requestfallbacks setting affect the returned model field and HTTP status?The GET /spend/logs endpoint returns request‑level spend data for analysis.
Extract the request‑level spend log from the proxy and pinpoint the keys and models responsible for recent spending.
model_group column to see which model actually answered each requestclaude-haiku while the caller asked for gpt-3.5-turbomodel_name assigns a custom label (e.g., bulk-classify) to a provider model.
Expose two model groups by task name instead of by model name, so an application asks for "bulk" or "judgment" and the config decides what actually answers.
litellm_params"model": "bulk-classify" and another using "model": "judgment-review" in the same JSON shape as previous lessonsmodel_name is just a label you choose, so naming it by task lets you swap underlying models later without changing any caller codemodel_name like bulk-classify in config.yaml change the way callers specify tasks versus concrete provider models?The /health/readiness endpoint reports the proxy’s readiness status.
Turn your local LiteLLM container into a network‑accessible service that survives failures.
/health/readiness endpoint returns 200 OK/health/readiness after container deployment?12 outcomes in all — one per recipe below.
Multiple AI models behind one URL
LiteLLM can run as a Dockerized proxy that normalizes API calls to different LLM providers. By defining each provider in the config file you expose a single endpoint that forwards requests to the chosen backend, simplifying client code.
Want to limit API spend
The LiteLLM UI lets you generate API keys (virtual keys) with per‑key quotas. When a user exceeds the allocated request count or token budget, LiteLLM returns HTTP 429 and logs the overage, enabling precise cost control.
Need a local LLM API you can call
Docker provides an isolated environment for LiteLLM, ensuring all dependencies are met and the service runs consistently across machines. Starting the container exposes the API on localhost:4000, ready for configuration.
Want to add a new AI model without touching code
The dashboard lets you register external LLM providers without editing code. Selecting a provider, entering its API key, and saving creates a model entry that LiteLLM can route requests to.
Need to define multiple models without manual entry
A configuration file allows bulk model definitions and version‑controlled setup, which the Docker container can read at startup to auto‑populate models.
Need to stop AI costs from exploding
Virtual keys let you grant scoped access to specific models and enforce spending caps, preventing runaway costs. The dashboard captures these constraints per key.
More than two requests per minute hit the limit
Rate limits protect APIs from overload. LiteLLM enforces per‑key RPM/TPM caps and records each request with metadata, enabling audit trails.
I need a local OpenAI‑compatible API
The video shows how to clone the LiteLLM repo, set a master key and salt in .env, then launch the proxy with Docker Compose. This creates a local HTTP server (default port 4000) that serves the OpenAI‑compatible API for all configured models.
Need to call both OpenAI and Groq models
After the proxy is running, the admin UI lets you register multiple LLM providers. By entering each provider’s API key and selecting a model (e.g., GPT‑4.1 mini for OpenAI, Llama 3 for Groq), the gateway can route requests to either backend using the same OpenAI‑compatible endpoint.
Need a team API key that can’t overspend or see other models
The UI’s Virtual Keys page lets you generate per‑team secret keys that are scoped to specific models and can enforce a dollar budget. This isolates teams (e.g., Finance) so they only see the Groq model and cannot exceed $10 spend.
Budget runs out after repeated requests
When the Finance virtual key’s $10 budget is exhausted, LiteLLM returns HTTP 429 Too Many Requests. The video demonstrates sending repeated requests until the limit is hit and then checking the logs for the exact spend and offending request.
The gateway records each request with model, token usage, team, and API key identifier. The video shows navigating to Logs in the UI to see a table where you can filter by virtual key or team and see exact token counts and dollar cost.
The same set on /recipes, filtered by tool and role.
Ask, share, or report — over on the Heidelberg AI community forum.