runtime · mostlygeek (GitHub handle; open-source, community-maintained)
llama-swap
A lab GPU rarely has room to keep five models loaded at once — llama-swap is the small, single-purpose fix: a zero-dependency Go proxy that sits in front of llama.cpp, vLLM, or any OpenAI/Anthropic-compatible server, and starts, stops, or swaps the right backend process based on the `model` field in the incoming request. A YAML file maps each model name to the shell command that runs it; ask for a model that isn't loaded and llama-swap starts it (stopping another if VRAM is tight), with TTL-based auto-unload and 'groups' for the models you do want to keep resident together. It doesn't make any engine faster — it makes a single GPU behave like it's serving a whole model catalogue instead of one fixed model, which is exactly the gap between a personal Ollama box and a lab that wants to offer five different models without five GPUs.
What it can help with
- model hot-swap
- gpu model switching
- dynamic model loading
- vram management
- backend proxy
- yaml configuration
- auto unload