One key, many models, and a budget that actually holds
A proxy that speaks one API to your applications and many APIs to providers, adding auth, budgets, routing, failover and logging in between.
The EU-sovereign inference chapter answers whose GPUs run your tokens. This one answers what happens to a request on the way there: which model it lands on, whose budget it spends, what happens when that provider is down, and whether anyone can reconstruct it afterwards. → A gateway is the single most useful piece of infrastructure for an organisation with more than one AI project, and it is one container. → It is also where the governance policy you wrote in an earlier chapter stops being a document and becomes a rule that returns HTTP 429.
One endpoint in front of every model you use — local, EU-hosted, frontier — with per-key budgets, provider failover and a log of every token. It is the layer where "we have a policy about AI spending" becomes a 429 instead of a memo.
Two or more projects, or two or more providers. One app talking to one provider does not need this.
Virtual keys per team, a spend ceiling per key, a routing policy, a fallback chain, and a request log with token counts.
Both anchors are open source and self-hostable; both also sell a managed tier. Self-hosted costs one small container and a database.
A gateway is on the critical path of every AI call you make. Budget for its failure modes before you route production through it.
The LiteLLM proxy (or openziti llm‑gateway) provides an OpenAI‑compatible /v1/chat/completions endpoint that forwards requests to configured providers.
Run a gateway exposing one /v1/chat/completions URL and forward calls to multiple model backends
/v1/chat/completions endpoint simplify integration with multiple model providers?A virtual key is an API token generated for a team that includes a max_budget enforcing a hard spending ceiling.
Generate a virtual key for a team and set a maximum budget that returns an error once the limit is reached
/key/generate on the LiteLLM admin endpoint, including your master‑key in the Authorization: Bearer header and a JSON body with {"models": ["gpt-4"], "max_budget": 50}LiteLLM_VerificationTokenTable backed by Postgresmax_budget, observe an HTTP 400 error with the message Authentication Error, ExceededTokenBudget…max_budget?The fallback chain maps a primary model to an ordered list of alternate models used when the primary fails.
Declare a fallback chain so a provider outage or rate‑limit becomes a slower response instead of a failed one, with zero changes on the calling side.
fallbacks entry under litellm_settings, mapping the primary model to an ordered list of alternates, e.g. fallbacks: [{"gpt-4": ["claude-opus", "local-llama"]}].content_policy_fallbacks, context_window_fallbacks and default_fallbacks for specific failure types.num_retries and cooldown_time in the same file to control how many attempts are made before moving on and how long a failing model is skipped.The Logs UI provides controls such as Date filter, Key column with Group by key, and Model column with Group by model to trace spend.
Pinpoint which API key, team and model caused an unexpected increase in spend using the gateway’s request log
Model groups like bulk-classify and judgment-review are named mappings that the gateway resolves to specific underlying models.
Route high‑volume tasks to a low‑cost model while reserving the frontier model for judgement‑heavy requests without changing application code
bulk-classify that maps to a cheap, high‑throughput model (or local open‑weight model)judgment-review that maps to the frontier model you want for judgement tasksmodel: "bulk-classify" and one with model: "judgment-review" – are resolved by the gateway to different real models and show different per‑call costsbulk-classify and judgment-review determine which underlying model processes each request?Deploying multiple gateway instances behind a load balancer with datastore backups ensures high availability.
Define an availability strategy for the AI gateway before routing production traffic through it
The decision framework evaluates whether you have two or more projects or providers before justifying a gateway.
Make a concrete decision about deploying a gateway using the two‑or‑more projects or providers threshold
30 outcomes in all — one per recipe below.
Finance or coding queries go to the right AI model
Semantic routing lets you define topics with example utterances and map each topic to a specific LLM model. When a request arrives, the gateway matches the prompt against the topics and forwards it to the mapped provider, ensuring cost‑effective and accurate responses.
The gateway exposes a single HTTP endpoint that abstracts away individual API keys and URLs for each LLM provider. Developers authenticate once, and the gateway handles authentication to the underlying models, providing high availability via automatic fallback.
When your LLM usage hits its token budget
A token‑rate‑limit policy lets you set maximum tokens per time window for each client or business group. The gateway tracks usage and rejects requests that would exceed the limit, turning budget overruns into HTTP 429 errors.
Primary language model fails
If the primary model for a topic cannot be reached, the gateway can automatically route the request to a secondary model defined in the fallback configuration, ensuring uninterrupted service.
Want to use different AI models for each client region
Header‑based routing lets you route requests based on custom HTTP headers such as region or client ID. This enables scenarios like EU‑specific models for compliance or per‑customer model selection.
Want to connect a new language model to your gateway
An Agent Gateway backend is a custom Kubernetes resource that represents an LLM provider (e.g., Gemini, Anthropic). By creating a secret with the API key and applying a Backend CRD, the gateway can route requests to that model without changing application code.
Want one endpoint to call many AI models
Agent Gateway’s HTTPRoute can contain multiple rules, each matching a URL prefix and rewriting it before forwarding to a chosen backend. This lets one endpoint serve many models (e.g., /ai/gemini → Gemini, /ai/claude → Anthropic).
Local testing can’t reach the gateway
When using a local Kind cluster without an external load balancer, `kubectl port-forward` can forward the gateway service’s port to localhost, allowing curl or client apps to reach the gateway as if it were a public endpoint.
Need an API gateway running in my Kubernetes cluster
Agent Gateway provides two Helm charts: one for its custom resource definitions (CRDs) and another for the control‑plane deployment. Installing both sets up the gateway API objects and the data‑plane proxy in a Kubernetes cluster.
Need to limit which gateways can be created
A GatewayClass resource defines the type of gateway (e.g., Agent Gateway) that can be instantiated in the cluster. By creating a specific class, you enforce policy such as replica count or load‑balancer type for all Agent Gateways.
Need to swap AI providers without changing code
The `completion` function from LiteLLM abstracts away provider-specific SDKs. By passing the model name and a standard message payload, you can call OpenAI, Anthropic, Google Gemini, Grok, etc., with identical code.
Primary LLM fails
LiteLLM’s `fallbacks` parameter lets you list secondary models. If the primary model raises an error (e.g., 403 or timeout), LiteLLM automatically retries with the next model, ensuring uninterrupted service.
Can’t tell what an LLM call spends
LiteLLM maintains a pricing database and provides `completion_cost` which returns input tokens, output tokens, and monetary cost for the request, enabling per‑team budgeting.
Repeated LLM prompts feel sluggish
Enabling `cache=True` with `cache_type="local"` stores request/response pairs in RAM. Subsequent identical prompts hit the cache instantly, saving latency and token cost.
Coding questions cost too much
Using LiteLLM’s `router` you map abstract task names (e.g., "fast", "code") to concrete provider configurations. The router selects the appropriate model at runtime based on the task label.
My requests hit rate‑limit errors
Define several entries for the same provider with different API keys; LiteLLM’s `simple_shuffle` strategy rotates requests, automatically avoiding rate‑limit errors and balancing load.
Want one simple spot for all LLM calls in your chain
LangChain provides `ChatLiteLLM` wrapper that accepts the same config as LiteLLM. You can build a prompt template, chain it with other components, and still benefit from routing, fallbacks, cost tracking, etc.
Each app has its own hard‑coded AI model URL
The AI Gateway acts as a reverse proxy that presents one API endpoint to all applications, handling request transformation and routing to any configured LLM provider. This removes hard‑coded model URLs and SDK differences, simplifying integration.
Don’t know which team is burning AI tokens
The gateway records token counts for every request, tagging them with the originating application or team. This visibility lets finance and engineering pinpoint budget hotspots and enforce spend limits.
High AI costs on all queries
By inspecting request payloads (e.g., length, complexity) the gateway can route cheap queries to a low‑cost LLM and send demanding ones to an expensive, high‑performance model, reducing overall spend while preserving quality where needed.
Missing API key or PII in LLM calls
The gateway can enforce API‑key validation, PII masking, and access control before any request reaches an LLM, turning governance documents into enforced runtime rules that return HTTP 429 when limits are exceeded.
Need to change AI model without rewriting code
Because applications talk only to the gateway, changing the underlying LLM provider is a configuration update in the gateway rather than a code rewrite across multiple services.
Multiple AI providers but only one endpoint
LiteLLM acts as a single proxy that normalizes API calls across providers, letting you route requests through one endpoint. Installing it (self‑hosted or via Elestio) gives you immediate control over all your keys and models.
Need to call different AI models from one app
Within LiteLLM you register each provider as a “model” with its own credentials. The gateway then forwards calls to the correct upstream API based on the model selected in the request.
Need to cap a dev team’s monthly budget and request speed
Teams group users and can be assigned spending caps (daily/weekly/monthly) and token‑rate limits. This enforces governance policies automatically, returning HTTP 429 when a limit is exceeded.
Need to hand out LLM credentials safely
Virtual keys are scoped to a team and optionally to specific models, letting you hand out credentials without exposing real provider keys. They inherit the team’s budget and rate limits.
Can't tell which AI provider costs less
The built‑in analytics tab logs every request with model, token count, and cost. By filtering per team or key you can see which provider is cheaper for your workload and adjust routing accordingly.
Want different AI requests to hit the right model
A gateway sits between your app and external LLM APIs, letting you decide per request which model to use based on task type or cost constraints. By configuring routing rules in the gateway, you can automatically send translation jobs to a specialized model and cheap chat to a smaller one.
Need to enforce a spend cap on AI requests
The gateway can inspect each outgoing request, look up the cost of the target model, and reject or downgrade requests when a predefined budget is exceeded. This turns your governance policy into an enforceable rule that returns HTTP 429 when spending caps are hit.
My main LLM stops responding
By configuring secondary endpoints, the gateway can detect failed calls (timeouts, 5xx) and transparently retry the request against an alternative vendor or region, preserving uptime without code changes in the application.
The same set on /recipes, filtered by tool and role.
Ask, share, or report — over on the Heidelberg AI community forum.