Many models, one GPU
The first wall every home lab and small server hits
1Overview
A supervisor in front of one or more inference engines that decides which model is resident on the GPU right now, and swaps as requests arrive.
One model fits. The second one does not, and the failure is not graceful — it is an out-of-memory error halfway through somebody else's request. → This chapter is about sharing a single GPU between several models: loading on demand and unloading when idle, running models as OCI images so they behave like everything else in your stack, and knowing the point at which you should stop hand-rolling and move to a scheduler. → It is the operational chapter that sits between "I self-host a model" and "my team depends on the models I self-host".
Sharing one GPU between several models without the second one taking down the first. On-demand load and unload, models as OCI images, a warm-model policy, and the honest signal that you have outgrown a single box.
When you need a chat model, an embedding model and a coding model on hardware that fits one of them at a time.
A proxy with a model registry, an idle timeout, a cold-start budget, and a way to pin the model that must never be swapped out.
Free and open at every layer, including the Kubernetes option. The real cost is the cold start you pay whenever the wrong model was resident.
Swapping is not free. A policy that thrashes between two models will be slower than either one alone, and it looks like a hardware shortage.
2Lessons 8
2.1 Detect when multiple models compete for GPU memory
A CUDA allocation error appears in the server logs when VRAM is exhausted by loading more than one model onto the same device.
Identify GPU out‑of‑memory errors caused by loading more than one model onto the same device
- Start the first chat model server and confirm it runs successfully
- Attempt to start a second model server (for example, an embedding or coding model) on the same GPU
- Check the server logs for a CUDA allocation error indicating out‑of‑memory
- You'll see An error message in the server log indicating a CUDA allocation failure
- Takeaway VRAM is a hard reservation so two models cannot coexist without an explicit eviction policy
- Check How can you confirm that a CUDA allocation error in the server log indicates multiple models are competing for GPU memory?
- Cost Free to understand — this is a hardware constraint, not a product to buy.
2.2 Swap models on demand with llama‑swap
The llama‑swap proxy reads config.yaml entries, starts and stops model servers on demand, and presents a single OpenAI‑compatible endpoint.
Run llama‑swap so a chat request and an embedding request each get the correct model loaded without manual server management.
- Clone the repository using git clone https://github.com/mostlygeek/llama-swap.git.
- Open config.yaml in your favourite editor and add entries under
models:with each model name and itscmd:line, using${PORT}where needed. - Start the proxy by running docker compose up -d (or python -m llama_swap if you prefer a direct run).
- Send an OpenAI‑compatible request to the proxy URL (for example with curl) and observe the startup delay for the first model.
- Repeat the request for the same model and see the immediate response, confirming the model remains loaded.
- You'll see The first request for a previously unloaded model pauses while llama‑swap starts its server, then subsequent requests to the same model return instantly.
- Takeaway llama‑swap lets you declare a start command per model and handles all starting, stopping and routing behind a single endpoint
- Check What steps demonstrate that llama‑swap routes each request to the correct model without manual server handling?
- Cost Free and open source. The only real cost is the swap latency itself, covered in a later lesson.
2.3 Unload idle models automatically
A ttl (time‑to‑live) entry defines how many seconds an idle model remains loaded before llama‑swap automatically unloads it to free VRAM.
Set a ttl so an idle model releases its VRAM automatically instead of staying loaded forever.
- Open the llama-swap configuration file (e.g.,
config.yaml). - Add or modify a
ttl:entry under each model definition with the desired number of seconds. - Save the file and restart the llama-swap service to apply the new settings.
- Verify that models unload after the specified idle period by monitoring GPU memory usage.
- You'll see A model that answered instantly an hour ago now takes several seconds to answer again because llama‑swap unloaded it after the TTL expired.
- Takeaway ttl is the knob that turns "first model loaded wins forever" into "only actively used models occupy GPU memory" — set it per model and expect latency trade‑offs
- Check How does setting a ttl value in the llama‑swap configuration affect a model’s presence in GPU memory after it becomes idle?
- Cost Free to configure. The cost is entirely in cold-start latency, tunable by how you set each ttl.
2.4 Pin a model so it never swaps
A persistent: true and swap: false group in llama‑swap.yml marks the listed model as always resident on the GPU.
Keep the chosen model loaded at all times while other models swap as needed
- Open the llama-swap.yml configuration file in your editor
- Add a new entry under
groups:with a name of your choice, setpersistent: trueandswap: false, then list the embedding model ID undermembers: - Create another group for the remaining models, set
swap: true(default) and optionallyexclusive: false, then list the chat and coding model IDs undermembers: - Save the file and restart the llama-swap service to apply the new groups
- You'll see The embedding model responds with consistently low latency, while chat and coding models only incur cold‑start delay on their first request after being idle
- Takeaway A
persistent: true+swap: falsegroup tells llama‑swap to never evict that model - Check Which configuration elements ensure that a chosen model never gets evicted while other models are allowed to swap?
- Cost Free to configure. The cost is VRAM permanently reserved for the pinned model, unavailable to anything else.
2.5 Package a model as an OCI image
The ramalama pull, run, and serve commands package a model as an OCI artifact, making it manageable like any Docker container.
Run a model through ramalama so it deploys, updates and is stored the same way as every other container in your stack.
- Pull a model from a registry using ramalama pull.
- Start an interactive chat session with the model via ramalama run.
- Expose the model as a REST API endpoint with ramalama serve.
- Show locally stored models by executing ramalama list.
- Display currently running model containers using ramalama ps.
- You'll see
ramalama pullandramalama runbehave like Docker commands and the pulled model appears as an image alongside your other containers. - Takeaway ramalama makes a model an OCI artifact — pull, run, store and push it through the same registry and tooling as the rest of your containers
- Check What sequence of ramalama commands turns a downloaded model into an OCI image that can be listed alongside other containers?
- Cost Free and open source. Registry storage for the pushed model image is whatever your existing registry already costs you.
2.6 Generate deployment files automatically
The ramalama serve --generate=… option emits a quadlet, Kubernetes YAML, or Docker Compose file referencing the exact image being served.
Create a quadlet, Kubernetes or Compose manifest for a model using ramalama instead of writing YAML by hand.
- Run ramalama serve with the
--generate=quadletflag to emit a systemd‑managed container definition for the model. - Run ramalama serve with the
--generate=kubeflag to produce a Kubernetes YAML manifest for the same image. - Run ramalama serve with the
--generate=composeflag to create a Docker Compose file when you need a compose‑based stack. - Optionally add an output path after the type, e.g.
--generate=kube:/etc/containers/systemd, to write the file to a specific directory.
- You'll see A
.container(quadlet) or.yaml(Kubernetes/Compose) file appears in the chosen directory, referencing the exact image you served. - Takeaway
ramalama serve --generate=…creates deployment definitions directly from the image reference so they stay in sync - Check How does the
--generateflag on ramalama serve create deployment files that stay synchronized with the served image? - Cost Free. The generated file still needs the same review as any other manifest before it runs in production.
2.7 Identify GPU thrashing
A latency spike occurring when a request follows a different model on the same GPU signals policy‑driven thrashing.
Detect when model swapping is causing latency spikes rather than a hardware bottleneck
- Examine recent request logs for latency values
- Compare each request’s latency with the model that answered the preceding request on the same GPU
- Record patterns where a chat request after an embedding request is consistently slower than consecutive chat requests
- Group models that frequently interleave and either pin the dominant one to a persistent group or allocate a second GPU for the pattern
- You'll see Latency spikes that line up with requests that follow a different model on the same GPU
- Takeaway Correlate slow requests with the previously served model to spot policy‑driven thrashing instead of assuming more hardware is needed
- Check What pattern in request latency logs indicates that model swapping is causing GPU thrashing rather than a hardware limitation?
- Cost No tool cost — this is a traffic-pattern diagnosis. The cost, if unfixed, is paid in latency on every request.
2.8 Move from single‑GPU serving to a multi‑GPU Kubernetes stack
Deploying the llm‑d/llm‑d helm chart with OCI images generated by ramalama and Kubernetes manifests creates a multi‑GPU inference stack.
Shift your inference workload onto multiple GPUs using the llm‑d serving stack
- Verify that your current OCI images built with ramalama are up‑to‑date
- Generate Kubernetes manifests with the
--generate=kubeflag for those images - Deploy the llm-d/llm-d helm chart (or equivalent) to a cluster with multiple GPUs
- You'll see Requests continue to queue behind swaps even after the pinned model and TTL settings have been optimised
- Takeaway When a well‑tuned single‑GPU policy still thrashes, the signal is to adopt a multi‑GPU Kubernetes solution such as llm‑d
- Check Which actions transition a single‑GPU serving setup to a multi‑GPU Kubernetes deployment using the llm‑d stack?
- Cost Free and open at every layer covered in this chapter, including llm-d. The real cost of graduating is operational — more nodes, more to monitor — which is exactly why it should wait for the honest signal, not precede it.
3See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.