The gateway on your bench
The small server in the middle that every app talks to
1Overview
LiteLLM on localhost:4000, serving two model names to every app on the bench.
LiteLLM runs on the bench at port 4000 and is the reason one switch changes the AI for everything. This chapter looks at it directly: ask it what it serves, send it a request by hand, and read the config file the dashboard rewrites. It is the same software an institution puts in front of its own models, one size down. → `litellm` and `llm-gateway` are the server-side versions, with budgets, keys per team and logging.
A gateway is one address that speaks the OpenAI API and forwards to whatever really answers. Every app on the bench is configured against it, once, and never again. It is worth meeting directly because the pattern is everywhere: the same LiteLLM sits in front of a university's GPUs, a company's Azure deployment, or five vendors at once.
Any time you want to know what the apps are really talking to, or to test a provider before switching to it.
litellm/config.yaml is generated from your provider choice; litellm/provider.json is what the dashboard displays.
The master key lives in the .env in your bench folder. It is a local secret, but it is a real one — do not paste it into a chat.
2Lessons 3
2.1 Ask the gateway what it serves
The gateway answers the OpenAI API, so every tool that speaks OpenAI speaks to it — curl included.
Get an answer from the gateway with no app in the way.
curl -s -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d '{"model":"default","messages":[{"role":"user","content":"Say hello in one word"}]}' http://localhost:4000/v1/chat/completionsin a terminal, with KEY set to LITELLM_MASTER_KEY from the .env in your HeidelbergBench folder.
- /v1/models lists what it serves:
defaultandhermes. - /v1/chat/completions is the same endpoint every app calls. If this works and an app does not, the problem is the app.
- That is the diagnostic ladder: gateway first, then the app. It takes ten seconds and saves an hour.
- You'll see A JSON response with a message from your own model.
- Takeaway When something AI-shaped breaks, ask the gateway first.
2.2 Read the file the dashboard rewrites
litellm/config.yaml maps a model name to a provider and a key. Choosing a provider on the dashboard rewrites this file from a template and restarts the gateway.
Do this first Ask the gateway what it serves
See the whole switching mechanism, which is smaller than it sounds.
- Open
litellm/config.yamlin your bench folder. Two entries: default and hermes. - Each has a model, a base URL and a key — exactly what you chose during setup.
- The template next to it carries placeholders like
__BASE_URL__. If you ever see one of those in the generated file, the gateway will start and answer nothing — the install test checks for exactly that.
- You'll see Under twenty lines that explain the entire provider switch.
- Takeaway The magic is a template, a substitution and a restart.
2.3 Watch the gateway's log while an app asks it something
Every app's request to a model passes through one container, heidelberg-litellm — its log is the one place that sees all of them.
Do this first Read the file the dashboard rewrites
See a request from an app hit the gateway, in real time.
docker logs -f heidelberg-litellmin a terminal, while the bench is running.
- Run the command above and leave the terminal open — it prints and keeps following.
- Switch to AnythingLLM (or any bench app) and ask it something.
- Watch the terminal. A new line appears for that request, naming the model it served, at the moment the app sends it — before the app has even finished answering.
- You'll see A new log line the instant you send the chat message.
- Takeaway The log is the fastest way to tell whether a stuck app ever reached the gateway at all.
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.