Serving a model to your whole team
Why the tool that got you started stops at user two
1Overview
A dedicated inference server that keeps a GPU busy across many simultaneous requests, exposed behind an OpenAI-compatible API.
LM Studio and Ollama are how everyone starts, and they are built for one person at a time. Put ten colleagues behind one of them and throughput collapses in a way that looks like a hardware problem and is not. → This chapter is about continuous batching: what an inference server does with ten simultaneous requests that a desktop tool does not, and how to measure the difference on your own box rather than trusting a benchmark blog. → vLLM and SGLang are the anchors, both under active daily development. The retired option is named too, because a lot of 2025 tutorials still point at it.
The jump from "it runs on my laptop" to "ten people use it at once". Continuous batching, KV cache sizing, and a benchmark you run yourself — because the gap between a desktop runner and a real serving engine under load is roughly an order of magnitude, and it does not show up with one user.
The moment more than one person, or one automated workflow, shares a model. Below that, a desktop runner is the right answer and this is overkill.
A serving engine, a batching scheduler, a KV cache budget, an OpenAI-compatible endpoint, and a load generator to prove the numbers.
Both engines are open source. The cost is the GPU and the electricity, which is why the benchmark comes before the purchase.
Measured under concurrent load, a desktop runner and a serving engine differ by roughly 19× on the same hardware. A team of ten behind the wrong one is not a hardware problem to solve by buying more hardware.
2Lessons 8
2.1 Compare a desktop runner with a serving engine
The benchmark summary compares latency and throughput between single‑user and 256‑user scenarios.
Identify how performance diverges when multiple users share the same GPU
- Read the provided benchmark description to understand the test conditions
- Observe the reported latency and throughput differences between one‑user and multi‑user scenarios
- Plan to run your own measurements on your GPU using a serving engine
- You'll see A benchmark summary showing near‑identical latency at one user and a 19× throughput gap at 256 concurrent users
- Takeaway Performance is similar for a single user but drops sharply as concurrency rises, so you must measure your own hardware
- Check What does the benchmark reveal about throughput as concurrency increases from one user to 256 users?
- Cost Free to read. The benchmark itself cost the author one A100 and a few hours; reproducing your own version is lesson 3.
2.2 Explain continuous batching and its impact on throughput
Continuous batching hands waiting requests to freed slots immediately, unlike naive batching which waits for the whole batch.
Show why continuous batching increases overall throughput without reducing single‑request latency
- Read the description of naive batching and note that the next batch starts only after every request in the current batch finishes
- Read the explanation of continuous batching and note that slots are handed to waiting requests as soon as any running request completes
- Compare the two throughput behaviours described under load and observe that continuous batching keeps throughput flat while naive batching drops
- You'll see A side‑by‑side comparison where naive batching’s throughput falls when a request runs long, while continuous batching keeps throughput flat
- Takeaway Continuous batching frees a finished request's slot immediately instead of waiting for the whole batch, which raises throughput under concurrent load and smooths out latency variance — but does nothing for a single request with nobody else in the queue
- Check How does continuous batching keep throughput flat compared to naive batching under load?
- Cost Free to understand — this is a scheduling algorithm, not a paid feature. Anyscale's writeup and vLLM's own blog post on the internals are both public.
2.3 Benchmark your own hardware — do not trust a blog post's number
GuideLLM (or vllm bench serve) performs load testing of a model endpoint to measure tokens per second and latency at various concurrencies.
Run a real load test against your model and GPU to obtain your own throughput numbers.
- Install and configure GuideLLM (or the
vllm bench servecommand) to target your OpenAI‑compatible endpoint. - Run a load test at concurrency 1, at an approximate team size, and at a higher level using the chosen tool.
- Compare the tokens/second and latency metrics across those runs to locate where throughput stops scaling and latency rises.
- Validate that the test uses the same model, quantisation, and prompt/response lengths you will use in production.
- Save the test script and results for future re‑runs after driver updates or configuration changes.
- You'll see A table showing tokens/second and p50/p95 latency at several concurrency levels, with the point where performance plateaus highlighted on your hardware.
- Takeaway Load‑test your endpoint at multiple concurrencies with GuideLLM or
vllm bench servebefore trusting any published multiplier - Check Which metrics should you compare across different concurrency levels to identify where throughput stops scaling on your GPU?
- Cost Free tooling either way. The cost is GPU time to run the sweep — budget 15–30 minutes for a handful of concurrency levels on one model.
2.4 Run and compare model serving engines
vllm serve, sglang.launch_server, and llama-server are the launch commands that start GPU or CPU serving engines exposing OpenAI‑compatible endpoints.
Stand up a serving engine with its actual launch command and understand the role of the other two options
- Run vllm serve
in a terminal to start the default GPU server on port 8000 - Run python -m sglang.launch_server --model-path
to launch SGLang’s OpenAI‑compatible endpoint - Run llama-server --parallel 4 --model-path
to start llama.cpp’s CPU/consumer‑GPU server with continuous batching
- You'll see Each command prints a ready message and an OpenAI‑compatible base URL, confirming the model is being served
- Takeaway vLLM and SGLang are the actively developed GPU servers; llama.cpp serves CPU or consumer‑GPU models with continuous batching, while TGI is archived and should be avoided
- Check What output confirms that each serving engine has started correctly and is ready to accept OpenAI‑compatible requests?
- Cost All three are free and open source. The cost is GPU time and the format conversion if your model is not already in the shape a given engine expects.
2.5 Determine the concurrency limit of your model deployment
The KV‑cache size reported in the startup log defines how many concurrent sequences can be kept in GPU memory.
Identify how many simultaneous requests your GPU can serve before the KV cache is exhausted
- Start the server with the flag --gpu-memory-utilisation (or --mem-fraction-static for SGLang) set to a value below 1.0
- Read the startup log for the line that reports the KV‑cache size allocated from the remaining GPU memory
- Run a load test and watch the throughput; note the request count where it stops rising while GPU utilisation is still under 100 %
- If the ceiling is too low, increase --gpu-memory-utilisation provided the card has free memory
- Otherwise reduce --max-model-len to match the longest prompt you actually need
- Optionally lower max_num_seqs (or the equivalent concurrent‑request setting) to limit how many requests share the KV cache
- You'll see The startup log shows the calculated KV‑cache size and, under load, request throughput stops increasing while GPU utilisation stays below 100 %
- Takeaway KV cache is a fixed memory pool that caps concurrency; tune --gpu-memory-utilisation or --mem-fraction-static, limit context length and concurrent‑request settings to stay within that budget
- Check Which log line indicates the allocated KV‑cache size, and how does it relate to the point where throughput stops increasing?
- Cost Free — this is a memory budget, not a paid resource. The cost is buying more VRAM if the ceiling is genuinely too low for your team.
2.6 Run your own model behind an OpenAI‑compatible endpoint
OPENAI_BASE_URL, model name, and an arbitrary API key are the environment settings required to route OpenAI client calls to a local server.
Serve a model that existing OpenAI‑client integrations can use without changing their code.
- Set the environment variable OPENAI_BASE_URL to
http://.:8000/v1 - Set the environment variable or client argument for the model name to the identifier used when launching the server.
- Provide any non‑empty string as the API key when initializing the OpenAI client.
- You'll see The same client code returns a completion from your server after only the base URL is changed.
- Takeaway Because vLLM and SGLang implement the OpenAI chat‑completions API, pointing tools at the new base URL and model name makes them work without further changes
- Check What minimal configuration changes allow an existing OpenAI client to obtain completions from your locally served model?
- Cost Free — no new integration work if the calling code already uses an OpenAI-compatible client, which is most current tooling.
2.7 Spot outdated deployment guides
An archived badge on a GitHub repository and the date of its latest commit indicate that the deployment guide may be outdated.
Identify when a tutorial recommends a serving stack that is no longer maintained before you use its launch command
- Open the GitHub repository URL for the serving stack you plan to use
- Check for an archived badge or banner on the repository header
- Review the date of the most recent commit shown on the main page
- You'll see The GitHub repository page displays the “This repository has been archived by the owner” banner next to the project name
- Takeaway Always verify a project's repository status before adopting its deployment instructions
- Check Which two indicators on a GitHub repository page reveal that a serving stack’s deployment guide is no longer maintained?
- Cost Free — a repository status check costs nothing and takes less time than reading the tutorial that needed it.
2.8 Choose the right serving approach for your team
--tensor-parallel-size is a configuration flag used when provisioning a GPU box for a dedicated serving engine.
Decide whether to keep using the desktop runner or adopt a dedicated serving stack based on measured load
- Run the load‑test from lesson 3 on your current setup and record the latency versus concurrency table
- Compare the recorded latency curve to your real‑world concurrent user count to see if it remains flat or bends upward
- Choose to stay with the desktop runner if latency stays low, or provision a GPU box and configure a serving engine (e.g., set --tensor-parallel-size) if latency rises at your actual concurrency
- You'll see A load‑test table showing latency staying flat at your real concurrency indicates staying with the desktop runner; a rise in latency at that concurrency signals the need for a serving solution.
- Takeaway Use the desktop runner until your own load test shows queueing at actual usage, then migrate
- Check Based on your load‑test latency curve, what condition tells you to switch from the desktop runner to a dedicated serving engine?
- Cost Free to decide. The cost you are protecting against is standing up and maintaining a serving stack nobody's queue actually needed yet.
3See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.