Keep a Shared Model Server Fast, Affordable and Running

T11-L05 · Run your own AI · Level 5 Operator · 22-minute read, plus 3 hours of doing

Monday, 09:10

In a lab it's six months after Lena moved her group's assistant onto a server. It worked so well that the whole department uses it now: five groups, 45 people. On Monday mornings it crawls. A question that takes five seconds on Friday afternoon takes a minute and a half at 09:10, and two group leaders have complained. The head of department has asked IT for a quote for a second graphics card. He has also asked Lena two questions she can't yet answer: "What does this cost us per month?" and "Who runs it when you're at a conference?"

In a company, Jonas's assistant now serves 40 staff through an EU model provider's API. Last month the provider had a three-hour outage, and the quality team couldn't draft anything that morning. The managing director has read that running your own model is cheaper and never goes down when someone else's does. She asks: "Should we rent our own GPU server?"

You are at Level 5 Operator. You run AI for everyone who relies on it, over months. When it's slow or down, work stops for dozens of people at once, and the decisions you prepare cost real money every month.

After this you can

• Watch the five signals that show a model server is healthy, and get an alert before users complain.
• Measure how many requests the server handles at once with a load test, and set a limit below the point where it fails.
• Choose between tuning the runtime, switching to vLLM, a smaller model and more hardware, from your own measurements.
• Compare the monthly cost per 1,000 answers of your server and an API, including your own time.
• Hand the server to a second operator with a runbook, an operations log and records an auditor can check.

Before you start

• Books: T01-L01 for the four-quadrant method. T11-L04 for the server this book keeps running: Caddy in front, Open WebUI, the model, backups and pinned versions. T03-L05 for the regression set, the fixed list of test questions a model must still answer correctly after any change.
• Admin access to the server, and an agreed window (an evening or weekend) when you may put test load on it.
• Your last three invoices or internal charges for the server, and a rough count of the hours you spend on it each month.
• A second operator, named by the service owner (the group leader, department head or manager who answers for the service), with two hours in their calendar.
• About 3.5 hours: 22 minutes to read, about 3 hours of doing, over two weeks.

Last verified 23 September 2026. Ollama, vLLM and GuideLLM are free. IONOS's AI Model Hub charges $0.17 per million input tokens and $0.71 per million output tokens for OpenAI's larger open-weight model. Hetzner's GPU server with 24 GB costs €214 a month before VAT. Details for experts at the end of this book.

The idea in one page

A model server shared by many people is capacity: a fixed amount of work per second that everyone draws on at the same time. Running it well comes down to four things.

1. Watch five signals. Each answers a question you'd otherwise hear from a colleague first.

An uptime checker is a small service that loads an address on a schedule and sends an alert when it fails. Time to first word (often called time to first token) is how long someone waits before the answer starts to appear. People tolerate a long answer; they don't tolerate a blank screen.

2. Know your envelope. The runtime (Ollama or vLLM) answers a certain number of requests at once. Beyond that, requests wait in a queue. Measure how many at once it can carry while people still see the first word quickly, and set your limit below that point.

The measurement is a load test: a tool sends 1, 2, 4, 8 or 16 requests at once, and you read the waiting times. Use the 95th percentile: the time within which 95 of 100 requests got their first word. An average hides the five colleagues who waited longest.

3. Know your cost per 1,000 answers. Add everything the server costs in a month: hardware or rent, power, backups, and your hours at your organisation's internal rate. Divide by the answers it gave, times 1,000. Do the same for an API, which charges per token (a piece of a word, roughly three quarters of an English word). Now you can compare. Often the API is far cheaper, and self-hosting is justified by something else: data that may not leave the building. That's a fine reason. Just name it, and put a price on it.

4. Make it survive you. A one-page runbook says what to do when each signal goes red. An operations log records every change, restore test, incident and access review with a date. A second operator has done a practice outage from the runbook. That log is also what you show an auditor: not a policy, but a dated record that the policy happened.

Who decides what: you measure, write the runbook and propose. The service owner decides the limit, the budget and whether to keep self-hosting. IT decides hardware and on-call cover. The data protection officer (DPO) decides how long logs that contain names or addresses are kept.

Worked example: two services, four quadrants

The method comes from T01-L01. Here it is for Lena's department server, then for Jonas's API-based assistant.

Lab: the Monday slowdown on a department server

Structure it. Lena: "45 people get the first word of an answer within ten seconds on a Monday morning, the department head knows what the service costs per month and what the alternative would be, and Tom can run it without me. A useful result: a capacity table, a cost sheet the department head signs, and a runbook Tom has used."

Understand it. She starts with what the server can already tell her. On Monday at 09:10 she runs nvidia-smi, NVIDIA's command that shows how busy the graphics card is and how much of its memory is used. The card is busy, and 9 of its 24 GB are free. So memory isn't the problem.

Then she reads Ollama's documentation on concurrent requests. By default, Ollama works on one request per model at a time and puts the rest in a queue. At 09:10 twelve people had asked something within the same minute. Eleven of them were waiting for the one in front.

That changes the question. Back to Structure it: "Before anyone buys a second card, find out how much this card can carry when it's allowed to work on several requests at once."

She also needs to know how many people really ask at once. She switches on Caddy's access log by adding one word, log, inside the site's braces in the Caddyfile. Caddy then writes one line per request into its container log, with the time and the address asked for. Every chat question goes to /api/chat/completions, so she counts those lines per minute. The busiest minute of the week had 12 requests; a normal busy minute had 5 or 6. The log holds colleagues' IP addresses, so she asks the DPO, who sets a retention of 14 days.

Choose tools.

• GuideLLM for the load test, a free tool from the vLLM project that sends a chosen number of requests at once to any server that speaks the OpenAI API format, the way of asking for an answer that OpenAI's API introduced and many servers copy, Ollama and vLLM included. It reports time to first word, answers per second and errors. The alternative is a short script of her own, but then she'd have to trust her own statistics.
• Uptime Kuma as the uptime checker, a free self-hosted monitor, run in one more container on a small IT machine rather than on the server it watches. A checker on the same server goes down with it.
• Ollama with more parallel requests as the first option, because it's one setting. vLLM as the second, a runtime built for many users at once: it slots new requests into the work already running on the card, instead of queueing them one after another. It needs more learning, and the group's model runs in both.
• A smaller model only if it passes the group's regression set from T03-L05. A smaller or more compressed (quantized) model needs less memory and is often faster, but "almost as good" has to be shown on the group's own questions. gpt-oss-20b, the open-weight model the group runs (one whose files anyone may download), already ships in a compressed form, so there's little left to gain here.

Build it: measure. IT agrees a test window on Wednesday evening. The server runs Ollama as in T11-L04, reachable only on the private network inside the server. For the test Lena adds one line to Ollama's block in compose.yaml, ports: ["127.0.0.1:11434:11434"], which makes the port reachable from the server itself and from nowhere else. Then she restarts it with docker compose up -d.

She installs GuideLLM on the server with pip install guidellm, Python's package installer. She runs the same test at 1, 2, 4, 8 and 16 requests at once. Each run lasts a minute, with requests shaped like the department's: a long prompt with chat history, a medium answer. One run looks like this:

guidellm run --backend kind=openai_http,target=http://127.0.0.1:11434 --profile kind=concurrent,streams=8 --constraint kind=max_duration,seconds=60 --data kind=synthetic_text,prompt_tokens=1500,output_tokens=400

The command sends 8 requests at a time, made of generated text of the stated length, for 60 seconds, and writes a table of the waiting times.

She runs the whole series three times. First with Ollama as it is. Then with Ollama allowed 4 parallel requests: she adds an environment: block to Ollama's service with the line OLLAMA_NUM_PARALLEL: "4" and restarts.

The third series, in a second window, uses vLLM on the same card. She stops Ollama with docker compose stop ollama, so the card is free, and starts vLLM once by hand, again reachable only from the server itself:

docker run --rm --gpus all --ipc=host -p 127.0.0.1:8000:8000 -v hf-cache:/root/.cache/huggingface vllm/vllm-openai:v0.30.0 --model openai/gpt-oss-20b

It downloads the model from Hugging Face, the public site where open-weight models are published, into the hf-cache volume, and answers on port 8000. GuideLLM's target changes to that port. When the series is done, she stops vLLM with Ctrl+C and starts Ollama again.

Her table, first-word time at the 95th percentile:

Every run finishes without errors, and nvidia-smi shows at least 3 GB free throughout. The second card isn't needed. The setting is.

Build it: decide the limit. The target is ten seconds at the busiest normal minute. Ollama at 4 meets it up to 8 at once, and the busiest minute on record had 12. vLLM meets it comfortably at 16. Lena chooses Ollama at 4 for now, because Tom already knows it and the change is one line, and writes the trigger for the next step into the runbook: if the log shows more than 8 requests in a minute on two days in one week, switch to vLLM, following the tested notes in the ops folder. She removes the test port line, restarts, and runs the regression set once to check the answers haven't changed.

Build it: watch. In Uptime Kuma she adds three checks. The first loads the assistant's HTTPS address every minute. The second is the same check's certificate option, which warns some days before the certificate expires. The third is a push check: Uptime Kuma gives it a secret address, and the last line of the nightly backup script calls that address with curl only when the backup worked. If no call arrives for 25 hours, the alert goes out.

Alerts go by email to her, Tom and the IT service desk. A small cron job mails her the free graphics memory and disk space every Monday at 09:15, from nvidia-smi and df -h, the command that shows how full each disk is.

Build it: cost. She collects the month's numbers. IT charges the department €380 a month for the GPU machine, including power, backups and patching. She spends about six hours a month on it, which the department values at €55 an hour, so €330. That's €710 a month. The log shows about 32,000 answers a month. So the department pays about €22 per 1,000 answers.

For comparison she prices the same work at an API: 32,000 answers with about 2,000 tokens in and 500 out each, at the list price of IONOS, a German cloud provider, for gpt-oss-120b, OpenAI's larger open-weight model, comes to about $22 a month in total, well under one dollar per 1,000 answers. The API is some thirty times cheaper.

But the institute's rule says unpublished results stay in the building, and the API is outside it. So the sheet she gives the head of department doesn't say "self-hosting is cheaper". It says: "Keeping unpublished work in the building costs us about €700 a month, or €22 per 1,000 answers. A second card would add to that and isn't needed at today's load." He signs it and sets a review in six months.

Build it: survive me. The runbook fits on one page, one line per red signal: Address down → check docker compose ps, restart the stack, and if it fails, restore last night's backup as in T11-L04. Slow → count this morning's requests per minute in Caddy's log and check nvidia-smi; if the trigger is met, switch to vLLM. Disk full → clear old model files with ollama rm. Each line names who acts (Tom, then IT's on-call service) and when to tell users, through the department mailing list. On Friday afternoon Lena stops Open WebUI without warning, and Tom handles it from the runbook while she watches. He gets stuck once: the runbook doesn't say where the backup files are. She adds the path. The practice outage goes into the operations log.

What it taught her: the second graphics card would have been thousands of euros spent on a problem that was one setting. The cost sheet turned a vague "it's expensive" into a price the department had chosen to pay.

Company: an outage and a question about renting a GPU

Structure it. Jonas: "The quality team can keep working when the provider is down, the managing director gets an honest answer about renting our own GPU server, and the contractor or Mia from quality can run the assistant without me. A useful result: a cost sheet she signs, a fallback we have tested, and a runbook."

Understand it. He starts with the bill. The provider's usage page shows tokens per day. Last month the 40 staff used about 45 million input tokens and 11 million output tokens across about 22,000 answers. At list prices for a comparable model, such as IONOS's for gpt-oss-120b, that's under $20 for the month.

Then the self-hosted option. A Hetzner GPU server with 24 GB costs €214 a month plus a one-off €209. It could run gpt-oss-20b, but not the larger model the team uses now, so it would first have to pass the team's regression set. Someone would patch, update and back it up: the contractor estimates eight hours a month at €90, so €720. Monthly total, with the setup fee spread over a year: about €950, or about €43 per 1,000 answers, against well under a dollar at the API. To break even, the team would need well over a million answers a month, some sixty times today's use.

And the outage question: a single rented server has no second copy either. Its outages would simply be his. Back to Structure it: "The question isn't which is cheaper; that's settled. It's what the team does when the provider is down, and how we notice first."

Choose tools.

• Stay with the current provider, under the existing contract.
• A second provider as a fallback, connected in Open WebUI next to the first, so that during an outage people pick the other model from the menu. Open WebUI can hold several connections at once. The contractor proposes IONOS, a German provider with a processing agreement (the contract on how a provider handles the company's data) the company can sign; another EU provider with a signed agreement would do as well. It costs nothing while it isn't used.
• Uptime Kuma on the contractor's monitoring machine: one check on the assistant's address, one on each provider's API.
• A spending limit on each provider key, with an alert at 80 %.

Build it. The contractor signs the fallback provider's processing agreement with the managing director. Jonas opens Open WebUI's Admin Panel, goes to Settings, Connections, and adds a second connection with IONOS's API address from its documentation and the new key. Only admins can see keys. He makes sure the IONOS model is visible to the whole team, not only to admins, so it sits in the model menu next to the usual one. He writes one line into the assistant record: If the main provider fails, choose the IONOS model in the model menu. Same rules for what you may paste. He tests it by switching off the main connection for ten minutes on a Friday afternoon; two colleagues switch models and carry on.

The runbook has four lines: address down, provider down, spending alert, certificate expiring. Mia from quality does the practice run: she switches the main connection off and on again, and finds the spending page. The contractor checks the certificate alert.

The cost sheet for the managing director is three lines: API about $20 a month, own GPU server about €950 a month, break-even above a million answers a month. Recommendation: stay, with a fallback. She signs it and asks for the same sheet once a year.

What it taught him: the outage felt like a reason to own the hardware. The numbers showed it was a reason to have a second road.

In both places the answer came from a measurement nobody had made before: how many requests arrive at once, and what an answer really costs.

What an auditor can check

Sooner or later someone asks you to show the service is run properly: an internal audit, the data protection officer, a customer's questionnaire, or your successor. Keep one folder, and keep it current:

• The service record from T02-L04 and T11-L04: what runs, where, for whom, with which data rules.
• The operations log: every update with old and new version, every restore test, every incident with times and what users were told, and a quarterly review of who has an account and who is admin.
• The capacity table with its date, and the limit and trigger you set from it.
• The cost sheet, signed by the service owner, with a review date.
• The runbook, and the date the second operator last used it.

Dated entries answer most audit questions on their own. A policy says what should happen; the log shows it did.

What goes wrong

The second card bought for a setting. Symptom: the service is slow at peak, and the answer is new hardware. Fix: check the queue and the card first. Ollama works on one request at a time unless you allow more; measure with more parallel requests before you buy.

An average that hides the wait. Symptom: the average looks fine, and people still complain. Fix: use the 95th percentile of time to first word, at the busiest normal minute.

The load test that becomes the incident. Symptom: colleagues' chats hang during your test. Fix: test in an agreed window, one step at a time, and stop at the first errors. Tell users in advance.

A monitor that dies with the server. Symptom: the server is down, and the alert never came. Fix: run the uptime checker on a different machine, and send alerts to at least two people.

A cost sheet without your hours. Symptom: self-hosting looks cheap because only the hardware is counted. Fix: add your hours, and your stand-in's, at the internal rate.

A smaller model chosen for speed. Symptom: answers get faster and quietly worse. Fix: any model change passes the regression set from T03-L05 before it goes live.

A runbook nobody has used. Symptom: on the day, the second operator can't find the backup or the password. Fix: a practice outage, from the runbook alone, twice a year and after every big change.

Do it yourself (3 hours, over two weeks)

1. Structure it (15 min). Write one sentence: how fast the service must feel at its busiest, what the service owner needs to know about cost, and who takes over from you. Ask the service owner to name the second operator.
2. Understand it (40 min). Turn on request logging if you don't have it, and ask the DPO how long it may be kept. After a week, find the busiest minute. At a busy time, look at the card's memory and the runtime's queue. Read your runtime's documentation on parallel requests. Rewrite your sentence if what you find changes the problem.
3. Choose tools (15 min). Pick the load-test tool, the uptime checker and the machine it runs on, and the options you'll compare (runtime settings, another runtime, a smaller model, more hardware).
4. Build it: measure (45 min). In an agreed window, run the load test at 1, 2, 4, 8 and 16 requests at once for each option. Write the table. Set the limit and the trigger for the next step. Run the regression set after any change.
5. Build it: watch and cost (35 min). Set up the three checks and the alerts. Fill in the cost sheet for your server and one API, per 1,000 answers, with your hours included. Take it to the service owner.
6. Build it: hand over (30 min). Write the one-page runbook. Stop the service unannounced in a quiet hour, and let the second operator handle it while you watch. Fix the runbook and write the practice into the operations log.

Exit check

Written finding: an operations pack for your shared model server.

• A capacity table from a load test at several levels of requests at once, with the 95th-percentile time to first word, errors and free memory, the limit you set and the trigger for the next step.
• A cost sheet: your server and one API, per 1,000 answers, with hardware or rent, running costs and your hours; the reason for self-hosting if it costs more; and the service owner's signature and review date.
• A one-page runbook covering the five signals, with who acts and when users are told.
• An operations log with at least the load test, one restore test and one practice outage.

It passes when the second operator has handled a practice outage from the runbook alone, the limit comes from the table rather than a guess, and the service owner has signed the cost sheet.

Rule to remember

Measure the queue before you buy the box.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T11-L04 · Put Your Team's Assistant on a Server Colleagues Can Reach: the server this book keeps running.
• T03-L05: the regression set every model change must pass.
• T12-L05: governance records and handover for any AI system a team depends on.
• T12-L04: the security check to repeat after big changes.
• T02-L05: running a shared assistant for a whole organisation.

Tools

• Ollama: Taught. The runtime in the lab example, and its parallel-requests setting.
• vLLM: Taught. The runtime for many requests at once.
• GuideLLM: Taught. The load test.
• Uptime Kuma: Taught. The uptime checker and alerts.
• nvidia-smi: Taught. Shows how busy the graphics card is and how much memory is free.
• IONOS AI Model Hub: Catalogued. The API in the cost comparison and the fallback.
• Open WebUI: Catalogued. Holds several model connections, which makes the fallback a menu choice.
• llama-swap: Catalogued. For switching between several models on one card.