Development & infrastructuremostlygeek (GitHub handle; open-source, community-maintained)Open source
llama-swap
Decision matrix
llama-swap runtime decision matrix
Overview
llama-swap is a small open-source proxy that starts the right local model server when a request asks for it and stops the others. It lets one GPU offer many models at one address. Use it once you already run llama.cpp or vLLM and have outgrown a single model.
Other good fits
Keeping stable model names for your tools while you swap the files and servers behind them.
Watching which model is loaded, with logs and a test playground in its web page.
+ Pros
Read pros
A GPU can usually hold only one or two models at a time. llama-swap solves that by sitting in front of your model servers. Your tools send every request to one address; llama-swap looks at which model is asked for, starts that model's server, stops the one that was running, and passes the request through. It works with llama.cpp's server, vLLM, whisper.cpp for speech, ComfyUI and any other OpenAI- or Anthropic-compatible server. A built-in web page shows what is loaded, streams the logs and has a playground for quick tests.
Pick it when
One shared GPU machine should offer a chat model, a coding model and a speech model to several tools.
You want your automations to ask for "chat-default" and be able to change the model behind that name later without touching them.
You already have working model servers and want to stop starting and stopping them by hand.
− Cons
llama-swap does not run models itself and does not download them; you prepare every model server first. Swapping trades memory for waiting: switching models costs load time, so it suits a handful of users better than a busy service. You set it up by editing a YAML file, not in an app.
Pick something else when
You want switching between models without writing a config: Ollama loads and unloads models on demand by itself.
You want one tool that downloads, serves and switches many model types: LocalAI.
One model must serve many users fast: run vLLM with that model permanently loaded.
Avoid if
You want model downloads and switching handled without writing a config file; Ollama already swaps models for you.
Answers must come back instantly; loading a different model takes seconds to minutes.
Requirements
Install llama-swap and prepare each backend independently first. A configuration maps model names to commands and paths; available memory, model files and compatible server flags remain your responsibility.
First run
Documentation-based suggestion — not run here.
Read first run
Two models behind one address. Read Your own AI server first if you have not set up a model server before.
Get one model working with llama.cpp's llama-server on its own.
Install llama-swap (Docker image, Homebrew, WinGet or a release binary).
Write a config.yaml with two entries, for example a small fast model and a larger careful one. Each entry is the llama-server command line with ${PORT} where the port goes, as in the example configuration.
Start llama-swap bound to localhost and open its web page.
Send one request that names the small model and one that names the large model, from the playground or from an automation.
It worked if both requests are answered from the same address and the web page shows the first model unloading as the second one starts. Expect the first request to each model to be slow while it loads.
Integration
The proxy exposes compatible routes and can use automatic backend ports. Its configuration can also define API-key behavior, timeouts and concurrency. Keep a local listener private unless you deliberately add access controls.
Check the result
Send one sample request using the configured model name and expect a response after the backend starts. A first request can be slow because the model is loading.
Links & evidence
Documentation-based · not hands-on · Sep 23, 2026
Rests on the llama-swap GitHub README, the example configuration and the releases page (v257), checked 23 September 2026. The curriculum team has not installed or run llama-swap for this review; the first project is a suggested exercise.
llama-swap README · Sep 23, 2026 — Hot-swaps inference servers on demand; works with llama-server, vLLM, whisper.cpp, stable-diffusion.cpp, ComfyUI and any OpenAI/Anthropic-compatible server; Docker, Homebrew, WinGet; MIT licence; web UI.
Example configuration · Sep 23, 2026 — Model entries with a cmd line per model, ports, timeouts and groups.
llama-swap releases · Sep 23, 2026 — Current release v257.
Further review notes & alternatives
Costs and operating effort
Prices checked 23 September 2026: llama-swap is free, open-source software (MIT licence) with no paid plan. You pay for the machine it runs on, usually one PC with a GPU.
The proxy itself is light. The work is in the config file: every model needs a correct command line, and a typo means that model fails to start. Update it along with the model servers behind it. It can listen on all network interfaces, so bind it to localhost or put it behind authentication if other people can reach the machine.
Alternatives and trade-offs
Ollama is simpler and swaps its own models, but only the ones Ollama can run.
LocalAI bundles many back ends and a model gallery in one server, with more moving parts.
LiteLLM routes requests between cloud and local providers, but does not start or stop local servers.
Switch model backends behind one compatible proxy endpoint.
Configure model startup and unloading with YAML commands.
Share one GPU across models without keeping every process loaded.
