Serving at scale

At Level 5 Operator, a model endpoint is shared capacity. A configuration that answers one prompt can still fail ten colleagues through queue growth, memory exhaustion, unfair admission, or costs nobody counted. Your job is to publish a measured operating envelope, reject excess work safely, and know when another box is the wrong answer.

2. Ten people arrive at once

The protected endpoint from T11-L04 works for one authorised user. At 08:00, ten Lab colleagues submit short literature-extraction requests together. In the Company framing, ten department users submit synthetic support classifications. The first answer is quick; later requests wait, clients retry, and the worker eventually runs out of memory. Someone proposes a second GPU because a one-request demonstration looked healthy.

That evidence cannot support a purchase. It does not state request shape, concurrency, completed throughput, queue time, tail latency, errors, model identity, runtime, memory headroom, or cost per useful request. It also confuses three different decisions: whether a model fits, whether a serving engine handles concurrent demand, and whether owning the service is cheaper than buying equivalent API outcomes.

This book replaces the single-request backend with a reviewed vLLM staging worker, holds the workload fixed, and runs bounded synthetic concurrency tests. You will compare only task-approved model variants, establish an admission limit below failure, and calculate all-in self-hosted and API cost per 1,000 successful requests. The Lab group's shared endpoint and the Company department endpoint use the same method, test shape, and report.

3. After this you can

• Serve one GPU to many concurrent users through a concurrency-aware engine, bounded admission, and one protected proxy.
• Choose a quantization by whether it fits and how fast it runs, after the same task regression gate has approved each candidate.
• Measure throughput, time to first token, end-to-end latency, errors, saturation, and headroom at representative concurrency.
• Place several model routes on one box without assuming that several simultaneously loaded servers will fit or remain fair.
• Say when self-hosting stops being cheaper than the API using the same successful-request denominator and the value of operator time.

4. Prerequisites

• T11-L04 - Your own AI server, including staging, production, TLS, individual access, private model ports, release records, rollback, and external verification.
• T03-L05 - Regression sets and evaluation ops. Its fixed task gate must approve the baseline and every quantized candidate before performance results are compared.
• An approved non-production staging GPU host, a named service owner, primary and substitute operators, and permission to run a bounded load test during an agreed window.
• A reviewed vLLM release compatible with the exact accelerator, driver, runtime, model architecture, and quantization format; GuideLLM in an isolated, version-pinned environment; and access to device and service metrics.
• Current local cost records and one approved API quotation covering the same region, model capability, input/output shape, and service requirement.

Use only public or synthetic prompts. The exercises use generated token shapes, not manuscripts, participant records, customer tickets, telemetry payloads, or copied production traces. Never point the benchmark at production, an endpoint you do not own, or a provider without written load-test permission. Agree maximum duration, concurrency, error count, spending, stop authority, and rollback before the first request.

5. The idea in one page

A serving engine turns waiting requests into scheduled work

A simple single-request loop processes one request while the GPU may have capacity for work from others. A serving engine such as vLLM schedules multiple sequences, batches compatible work continuously, and manages attention memory. That can increase aggregate throughput under concurrency. It does not create memory, guarantee fairness, make every model compatible, or make every user's response faster.

Measure four different outcomes:

Also record failures, queue depth or waiting requests, device memory, utilisation, power if available, and server restart count. Throughput can rise while p95 latency becomes unacceptable. An average hides the colleagues at the end of the queue.

Capacity is an operating envelope, not a peak number

Hold model, runtime, prompt tokens, output tokens, decoding settings, hardware, and test duration fixed. Increase only concurrency. A useful envelope is the highest tested level that meets the prewritten latency and error objectives with headroom. It is not the highest level that completed once.

Admission belongs before expensive model work:

individual identity -> per-group limit -> bounded queue -> vLLM scheduler -> one GPU
                         |                    |
                         +-- 429 + retry -----+-- metrics and alert

Use a finite in-flight limit, finite queue, maximum queue age, and caller-visible overload response. Retries need backoff and a cap; immediate automatic retries turn overload into more overload. Keep the T11-L04 proxy as the only public front door. The vLLM port, /metrics, profiling, model-loading, and other administrative routes stay private.

Quantization is a fit-and-speed candidate, not a quality adjective

Quantization stores or computes some model values at lower precision. For this capacity decision, compare only:

1. Does the exact artifact and runtime load with required memory headroom on the target device?
2. Under the same bounded workload, how fast does it run and where does it saturate?

Quality is not inferred from a format name. First run the unchanged 20-50-case gate from T03-L05; reject any candidate that fails its task or critical cases. Among candidates that pass, measure peak memory, TTFT, latency, throughput, errors, and compatibility. A smaller artifact may not be faster on particular hardware, and a runtime may fall back to an unintended execution path. Record the exact model revision, artifact hash, quantization method, runtime, kernels/backend, and device.

Several models need one residency policy

Starting three servers does not divide one GPU safely. Each may reserve model weights, runtime memory, caches, and request state. Choose deliberately:

• One resident model with aliases when tasks can use the same approved model.
• One resident model plus approved adapters when the runtime supports the exact adapter design and isolation requirements.
• Mutually exclusive residency through a gateway such as llama-swap: load one named model, unload it before another, and make cold-start delay explicit.
• Separate resident workers only when combined measurement proves memory headroom, fairness, isolation, and recovery.

Do not silently swap a consequential route to a different model. A route states its model, cold or warm status, concurrency, queue policy, owner, and overload outcome. If loading another model threatens the primary service, reject or schedule it rather than hoping the allocator will decide fairly.

Compare total cost for the same successful work

Normalize both options to 1,000 requests that pass the same regression and latency objective:

self-hosted monthly cost = equipment allocation + power + facilities + hosting
                         + support + software + monitoring + backup
                         + operator labour + expected downtime allowance

API monthly cost = input-token charge + output-token charge + platform charges
                 + storage/network/egress + support + integration labour
                 + monitoring + expected retry/failure cost

cost per 1,000 successful requests
  = monthly all-in cost / successful in-objective requests x 1,000

Use observed accepted outcomes, not theoretical tokens per second. Include paid and internal operator hours at an approved loaded rate. If demand is low, owned capacity remains paid while idle. If demand is high, API rates and limits may dominate. Price is still not permission: data location, capability, contracts, recovery, and exit requirements are separate gates.

6. The worked example: size the shared endpoint before buying

The Lab endpoint extracts structured fields from fictional literature notes for a research group. The Company endpoint classifies fictional department messages. Both already pass their own 24-case regression set. For capacity testing, both use synthetic text with the same token shape: 256 prompt tokens and at most 128 output tokens. Generated token-shaped input is suitable for infrastructure measurement, but not for claiming task quality.

Freeze the test contract

Create one capacity-report.md and record this before testing:

Environment: staging only; public model port: none
Framing and service: [Lab extraction | Company classification]
Proxy/release revision: [exact]
Baseline model/artifact/hash: [exact]
Candidate model/artifact/hash and quantization: [exact or none]
vLLM version/image digest; GuideLLM version; driver; device; host: [exact]
Workload: synthetic_text; 256 prompt tokens; 128 output tokens
Concurrency rows: 1, 2, 4, 8, 10
Per-row limit: 60 seconds; stop after 5 errors or operator stop
Illustrative local objective: success >= 99%; p95 TTFT <= 2.0 s;
  p95 end-to-end <= 15 s; no OOM/restart; memory headroom >= 10%
Admission proposal: maximum 8 in flight; queue maximum 8; queue age 30 s;
  overload HTTP 429 with Retry-After; clients make at most 2 backed-off retries
Rollback: restore the recorded T11-L04 staging backend revision
Test owner / observer / stop authority / window: [roles and approved values]

The numeric objective and concurrency proposal are teaching examples. Replace them with approved service requirements before observing results; do not loosen them afterward to make a purchase case. Ten concurrent users does not necessarily mean exactly ten simultaneous model requests, so later compare this synthetic pattern with privacy-safe aggregate production demand.

Put vLLM behind the existing boundary

Do not install a new runtime over the known-good production worker. Deploy a separate staging revision. Follow the official installation path for the target platform, pin the reviewed package or image and its resolved digest, and confirm the model license and provenance. A generic launch declaration is:

vllm serve "$MODEL_PATH" \
  --served-model-name "$SERVED_MODEL" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-model-len 4096 \
  --max-num-seqs 10

MODEL_PATH must resolve to the reviewed local artifact or approved registry revision; SERVED_MODEL is a stable staging route name, not a secret. The limits are a bounded starting configuration, not tuning advice. Verify current CLI syntax with vllm serve --help for the pinned release. Do not add --trust-remote-code unless separately reviewed; do not enable development, dynamic adapter-loading, profiling, or cache-reset endpoints on a shared route.

On the host, require curl --fail http://127.0.0.1:8000/health and curl --fail http://127.0.0.1:8000/v1/models to succeed. From outside, port 8000 must not answer. Update only the staging proxy route to allow the required completion path after individual authentication; retain denial of unapproved and administrative paths. Repeat the logged-out, authenticated synthetic request, TLS, port, restart, and rollback checks from T11-L04 before load testing.

Why is this better than the previous single-request worker? Do not answer from architecture alone. The pass evidence is the fixed concurrency matrix. Keep a one-request row as baseline, then show how completed throughput changes as several streams remain active.

Run a bounded concurrency matrix

Install a reviewed GuideLLM release in an isolated environment and record guidellm --version. Read guidellm run --help; the project evolves, so do not copy an unpinned latest container into an operating record. From the staging host, use the private loopback backend so benchmark traffic never opens a second public front door:

for streams in 1 2 4 8 10; do
  mkdir -p "results/c${streams}"
  guidellm run \
    --backend kind=openai_http,target=http://127.0.0.1:8000 \
    --profile "kind=concurrent,streams=${streams},warmup=0.1,cooldown=0.1" \
    --constraint kind=max_duration,seconds=60 \
    --constraint kind=max_errors,count=5 \
    --data kind=synthetic_text,prompt_tokens=256,output_tokens=128 \
    --output json "path=results/c${streams}/benchmarks.json"
done

Use the request format required by the reviewed model and pinned GuideLLM version; add the documented request_format backend setting if chat completions are required. Run one row at a time. Between rows, wait for requests and memory to settle, confirm worker health, and save device observations. Stop immediately for an out-of-memory event, worker restart, persistent error, abnormal temperature or power state, audit failure, impact outside staging, or the agreed operator stop.

For every row, copy from the machine-readable report into the single capacity report:

Never paste invented results into these cells. Validate the report's metric definitions and denominators against GuideLLM documentation. If concurrency 8 passes and 10 fails p95 latency or headroom, the demonstrated ceiling is 8, not 10. Publish lower admission if normal traffic variability, monitoring, restart, or another resident workload consumes the margin. Configure the proxy or approved gateway to enforce that bound and deliberately submit one excess synthetic request. It must queue only within policy or return the documented 429; it must not create an unbounded wait or restart the worker.

Compare quantized candidates fairly

Suppose the baseline nearly fills device memory. Select at most two reviewed quantized artifacts of the same task-approved model family that vLLM documents as supported on the exact hardware. Do not convert production material, and do not download a community artifact whose source, license, hash, and method are unknown.

For each candidate:

1. Run the unchanged T03-L05 regression set and critical cases. Reject failures before load testing.
2. Deploy to staging with its exact artifact and runtime record. Confirm the intended quantization implementation loaded without unexplained fallback.
3. Repeat the identical concurrency matrix. Change no prompt/output length, decoding settings, runtime version, or objective.
4. Record artifact size, loaded and peak memory, headroom, throughput, TTFT, latency, errors, and cold-start time.
5. Choose only among passing candidates. State the decision as “candidate Q fits with X observed headroom and meets the speed objective through concurrency Y,” never “4-bit is best.”

Use the regression report by reference inside the capacity report; do not submit it as another exit artifact. If no candidate passes both gates, retain the baseline, reduce admitted demand, select a smaller approved model through a new quality review, or investigate hardware. Quantization is not a waiver for a failed task.

Put two routes on one box without pretending it is two GPUs

The Lab may need extraction and short-answer routes; the Company may need classification and drafting. First ask whether one approved resident model can serve both aliases. If not, make a residency table:

Use a reviewed gateway such as llama-swap only as a control plane for explicit model switching; verify its current configuration and failure behavior. Test route identity, cold start, bounded queue, memory release, primary-route availability, and rollback with synthetic requests. A model name in a response is not enough: correlate route, loaded artifact, worker, and metric record. If both models must be resident, rerun the capacity matrix under combined demand. If they do not fit with headroom, schedule mutually exclusive windows or separate the workloads. Several isolated CPU workers can be appropriate for smaller GGUF models through llama.cpp or LocalAI, but they require their own measured envelope; they are not vLLM results by proxy.

Calculate cost per 1,000 successful requests both ways

Use one month and three demand scenarios: low, expected, and peak. Keep the same prompt/output token shape, task pass rule, and latency objective. Fill the report with current evidenced values. To rehearse the arithmetic only, use these explicitly synthetic values:

The arithmetic is 2,000 / 80,000 x 1,000 = EUR 25 and 2,400 / 80,000 x 1,000 = EUR 30. These are invented teaching inputs, not market prices. Replace them with invoices, measured power, approved depreciation/allocation, current provider pricing and token accounting, and recorded hours. Include patching, upgrades, incident response, evaluation, capacity work, backups, security review, procurement, and substitute training. “My time is free” is not an operating model.

Calculate the crossover transparently. If F_local is fixed monthly local cost, V_local is variable local cost per request, and V_api is API variable cost per request, then the simplified request crossover is:

N = F_local / (V_api - V_local)

Use it only when the denominator is positive and assumptions are comparable. Real costs can be stepped, tiered, committed, tax-sensitive, and capacity-limited, so show low/expected/peak scenarios rather than one false-precision answer. Self-hosting stops being cheaper whenever its all-in cost per 1,000 successful in-objective requests exceeds the API's for the relevant scenario. It can also stop being the right choice before that point if staffing, recovery, capability, support, or mandatory controls fail. Conversely, a cheaper API is ineligible if it fails a mandatory data or contractual gate.

The operator's recommendation must allow “stop self-hosting.” Trigger review when utilisation stays below the approved floor, operator hours exceed budget, required availability cannot be staffed, the API wins the like-for-like scenarios, hardware refresh is due, or a safe supported runtime is no longer available. Preserve export and rollback paths; do not keep a box merely to justify its purchase.

7. What goes wrong

Hardware is bought before demand is measured

Symptom: the purchase request cites model size and a one-user demo. Fix: block it until the fixed concurrency matrix, headroom, objective, growth scenario, and all-in comparison are reviewable. Measure the existing box first.

The benchmark sends one request at a time

Symptom: latency looks excellent while the real queue at 08:00 grows. Fix: keep concurrency 1 as baseline, then test representative bounded concurrent streams and report tail latency, throughput, errors, queueing, and saturation together.

Quantization is selected from a quality claim

Symptom: a smaller artifact is called “almost identical,” or a public leaderboard substitutes for the service regression set. Fix: gate task behaviour first. Among passing candidates, select by measured fit, speed, compatibility, and headroom on the target stack.

Several models reserve the same memory

Symptom: each model passes alone but combined workers restart or evict unpredictably. Fix: publish residency and yield rules, measure combined demand, and use one model, explicit switching, scheduling, or separate capacity. Never depend on accidental eviction.

Average latency hides overload

Symptom: mean latency passes while p95 rises beyond the user's wait or errors are excluded. Fix: report p50 and p95, successful and failed counts, TTFT, end-to-end latency, and queue outcome. Errors remain in the denominator.

The cost sheet omits the operator

Symptom: a hardware invoice is compared with a complete API bill. Fix: add facilities, power, support, monitoring, backup, downtime allowance, refresh, and every operating hour at the approved loaded rate. Apply equivalent integration labour to the API side.

There is no capacity headroom

Symptom: the selected limit is the first row before OOM. Fix: lower admission, retain explicit memory and latency margin, and rerun after model, runtime, request-shape, or colocated-workload changes.

Load testing becomes an incident

Symptom: production users slow down, costs exceed approval, or clients continue after errors. Fix: stop the generator, restore the known-good staging route, verify unrelated service health, preserve bounded evidence, and follow incident reporting. Never discover permission after generating load.

8. Do it yourself: a 120-minute capacity and cost decision

Minutes 0-15: choose Lab or Company, name test and stop owners, freeze the staging release, synthetic 256/128-token workload, concurrency rows, duration, maximum errors, service objective, admission proposal, and rollback. Confirm the task regression set and current baseline pass.

Minutes 15-30: deploy the reviewed vLLM worker to staging behind the existing proxy boundary. Record exact model, artifact hash, quantization, runtime, image/package, driver, device, and configuration. Verify private backend port, health, individual access, denied management paths, metrics access, and rollback.

Minutes 30-55: run concurrency 1, 2, 4, 8, and 10 sequentially with GuideLLM synthetic data. Observe requests, output tokens, p50/p95 TTFT and latency, errors, queueing, device memory, and restarts. Stop at the agreed boundary; never continue through OOM to complete a table.

Minutes 55-70: identify the highest row meeting all prewritten requirements with headroom. Set an admission limit no higher than it. Generate one controlled excess request and verify bounded queueing or 429 plus retry guidance, followed by healthy recovery.

Minutes 70-88: compare one task-approved quantized candidate, or document why none is eligible. Run the unchanged regression gate first, then the same capacity rows. Decide only from observed fit and speed among quality-passing options.

Minutes 88-100: write the one-box residency table for a second route. Test an explicit cold switch or combined residency only if approved. Require route identity, bounded waiting, memory headroom, primary-service protection, and rollback.

Minutes 100-114: calculate self-hosted and API all-in monthly cost and cost per 1,000 successful in-objective requests for low, expected, and peak demand. Cite dates and evidence for every input, include operator time, and state the crossover or why it cannot yet be calculated.

Minutes 114-120: have the substitute operator inspect raw results and reproduce one throughput and one cost calculation. Finish the single report with pass/fail, capacity limit, quantization decision, cost recommendation, owners, expiry date, and triggers to stop self-hosting. Remove temporary benchmark access and leave staging in its known-good state.

9. Exit check

Deliver exactly one artifact: one passing capacity-and-cost report containing the concurrency test's recorded successful-request throughput plus a like-for-like self-hosted versus API cost-per-1,000-requests comparison.

It passes only when the report identifies the exact model/artifact, quantization, vLLM and GuideLLM versions, hardware, test date, synthetic 256/128-token shape, fixed concurrency rows, prewritten stop limits and service objective; records attempts, successes, errors, requests/second, p95 TTFT, p95 end-to-end latency, and memory headroom for every completed row; shows a bounded overload result; and states the demonstrated admission limit. A stopped unsafe row remains visible as stopped, not silently omitted.

The same report must show low, expected, and peak demand; all-in cost categories including operator labour; successful in-objective requests as the shared denominator; arithmetic for both costs per 1,000; evidence dates; non-financial gates; the crossover or documented unknowns; and a keep, change, or stop-self-hosting recommendation with owner and review date. The referenced regression gate must pass for the selected model. Raw benchmark files, regression results, invoices, and metrics support this one report; do not submit them as additional exit artifacts.

10. Rule to remember

Measure the load before you buy the box.

11. Further reading & tools

• Taught: T11-L04 - Your own AI server - supplies the protected proxy, staging, release, rollback, and external-port boundary retained here.
• Taught: T03-L05 - Regression sets and evaluation ops - supplies the unchanged task-quality gate every model and quantization must pass.
• Taught: Serving a model to your whole team - defines bounded admission, queue age, overload outcomes, signals, and operator response.
• Taught: Make an AI model smaller - separates quantized candidate creation from task, fit, speed, and compatibility evidence.
• Taught: Many models, one GPU - establishes resource budgets, residency, route isolation, and recovery for colocated models.
• Taught: Sizing the box and Big memory, slow bandwidth - separate compatibility, capacity, headroom, and observed service behaviour.
• Taught: Local vs cloud: what it actually costs - provides the shared workload, useful-outcome denominator, scenarios, and all-in ledger used here.
• Taught · Primary: vLLM online serving (opens in a new tab) - official current endpoints, health, metrics, and production warnings.
• Taught · Primary: GuideLLM repository and documentation (opens in a new tab) - official CLI, synthetic workloads, concurrent profiles, constraints, metrics, and report formats; pin a reviewed release.
• Catalogued · Primary: vLLM quantization support (opens in a new tab) - current format and hardware compatibility; recheck for the pinned release and target device.
• Catalogued · Primary: vLLM GPU installation (opens in a new tab) - current NVIDIA, AMD, Intel, and Apple support paths; a device brand alone is not compatibility evidence.
• Catalogued: llama.cpp server (opens in a new tab), LocalAI (opens in a new tab), and llama-swap (opens in a new tab) - alternative serving and explicit model-switching paths that require their own measured envelopes.
• Catalogued: LLM Compressor (opens in a new tab) - candidate quantization tooling; conversion success does not replace regression and target-host measurement.
• Catalogued: Beyond NVIDIA and AI for robotics & edge devices - portability and edge control gates; never extend this load exercise to physical actuation.
• Catalogued · Primary: Google SRE: Handling overload (opens in a new tab) - load shedding, queueing, and protecting a service under demand.
• Catalogued · Primary: FinOps Framework (opens in a new tab) - current technology-cost and unit-economics practices for reviewing the all-in comparison.
• Catalogued: T02-L05 - Running the assistant for everyone - applies identity, group quota, attribution, and retention controls around the measured endpoint.
• Catalogued: Tools index - compare approved serving, gateway, benchmarking, and cost-analysis options after the workload is fixed.