Keep a Production Prompt Healthy for Months, and Hand It Over

T03-L05 · Prompting & context · Level 5 Operator · 19-minute read, plus 4 hours of doing

Nothing changed, and the numbers moved

The extraction pipeline has run every night for eight months. It now serves three review groups in the department, twelve reviewers in all, and every change to the prompt has gone through tests, a reviewer and a tagged release. Nobody touched it in October. Yet in the first week of November a reviewer finds a sample size of 120 on a paper that hasn't enrolled anyone yet. It's a trial protocol, a paper that describes a planned study, and the prompt took the planned number for a real one. A search finds 31 more. Two groups have already screened them into their reviews. And the PhD student who wrote the prompt leaves in December.

In a company the sorting prompt now routes 600 customer emails a day for twelve support staff in three countries. Purchasing wants the service moved to a model under the company's existing Microsoft contract, to save money and simplify invoices. The switch looks like a one-line change. If it quietly sends cancellations to the general queue, customers who wanted to leave get a sales reply instead.

You are at Level 5 Operator. When this prompt goes wrong, many people's work goes wrong with it, often for days before anyone notices. This book shows you how to keep it healthy over months: what to watch, what to do when it breaks, who takes over, and what you can show when someone asks.

After this you can

• Grow the test set into a regression set of 30 to 50 cases, with a critical group that must always pass, and add every real failure to it.
• Gate every change to the prompt, model or provider on a pass rule written before the run, and run the set every week even when nothing changed.
• Watch three signals in production, each with a named person who responds: did it run, has the shape of the output shifted, and does a weekly human sample still pass.
• Handle a quality incident: hold or roll back, find the affected outputs, tell the people who used them, and add the failure as a case.
• Hand the operation to a named substitute who has run a blocked release and a rollback without you, and keep the records an auditor asks for.

Before you start

• Books: T01-L01 for the four-quadrant method. T03-L03 for the test set and pass rule. T03-L04 for the repository, the automatic test run on every pull request, releases by tag and the rehearsed rollback. This book assumes all of that is in place.
• Promptfoo, the test runner, and the prompt's repository on your code server.
• The production logs your pipeline already writes: at least the date, the release tag and the output of every run.
• A substitute: a named colleague who will take over when you're away or gone, with three hours in their calendar over the next month.
• The people who decide: the group leader or service owner who approves releases, and, where outputs involve personal data, your data protection officer.
• About 4 hours 20 minutes: 19 to read, about 4 hours to do, over a month.

Last verified 23 September 2026. Promptfoo is free and open source; this book pins version 0.123.1. Anthropic gives at least 60 days' notice before retiring a model. Langfuse, which records every model call, is free to run yourself, but automatic deletion needs a paid licence. Details for experts at the end of this book.

The idea in one page

At Level 4 you made sure that every change passes a gate and can be undone. At Level 5 the question is what happens between changes, over months, and when you're not there. Four habits answer it.

1. The set grows from real failures. Your test set from T03-L03 held 15 cases, chosen in the spring. The inputs your prompt sees in November are different. So every confirmed failure in production becomes a case, anonymised if needed, within a week. The set becomes a regression set: cases that check that what once worked, and what once broke, still behave. Mark a critical group, the cases whose failure would harm someone or stop work. They must pass every time, whatever the overall score.

2. The gate covers every change, including the ones you didn't make. The gate is what T03-L04 built: the automatic test run and a reviewer that every change must pass. A new prompt wording, a new model, a new provider, a change to what feeds the prompt: each runs the full set against a pass rule written before the run. And the set runs once a week against production even when nothing changed. That catches the changes that happen to you: an expired key, a provider's update, a new kind of input.

3. Watch three signals, not a dashboard.

A shape shift is often the first sign. If the share of unknown drops sharply, the prompt may have started guessing. If one queue empties, emails are going somewhere else. Neither shows up as an error.

4. Everything you do, someone else can do. A runbook, a short page of the steps for each situation, sits in the repository. A named substitute has run a blocked release and a rollback from it without your help. And the records you keep anyway, releases, test results, weekly signals and incident notes, are the answer when an auditor, a journal reviewer or a customer asks how you know the output is right.

Worked example: a prompt that outlives its builder

Both versions follow the four quadrants from T01-L01. Follow the one closer to your work. The people, cases and numbers are examples written for this book.

Lab: the department's extraction pipeline

Structure it (15 min). Karim, the data manager, has run the extraction pipeline since the spring. Leonie, who wrote the prompt, leaves on 20 December. Karim writes: "The extraction prompt must stay right for twelve reviewers when its inputs change, when we change the model, and when Leonie and I are both away. A good result: we notice a problem before a reviewer does, we can say which rows it touched, and Sofia can run everything from the runbook." The group leader names Sofia, a PhD student from the second review group, as his substitute.

Understand it (60 min). He starts with the protocol incident. The test set has 15 cases, all abstracts of finished studies. It has no protocol paper, no conference abstract, no correction notice, no abstract in German. He counts what the feed actually sent in October: 9% protocols, 4% conference abstracts, a handful of corrections. So a quarter of the real inputs are kinds the tests have never seen.

Then he looks at what would have told him. The rows in the review database record the release tag, and every field. He runs a query for the share of rows where the sample size is unknown, week by week. From May to September it sat between 18% and 24%. In the last two weeks of October it fell to 9%. The warning was there for two weeks; nobody was looking.

He also collects every correction reviewers made since May: 214 of about 9,000 rows. He sorts them by cause. Most are one-offs. Three causes repeat: protocols, abstracts that give a sample size only in a table, and abstracts where the primary outcome is named in the title but not in the text.

This changes the problem. He'd thought of it as "fix the protocol bug and write a handover note". The real gap is that the tests follow what the group imagined in the spring, and nobody reads the numbers that show when reality moves. Back to Structure it: "The set must follow what the feed sends. Someone must read three signals every week. The fix for protocols goes through the same gate as everything else."

Choose tools (15 min).

• Promptfoo stays as the gate. A second workflow in the repository runs the set every Monday morning against the production release and posts the result to the group's chat channel.
• The signals come from the review database. A short query, run by the same Monday workflow, posts the week's numbers: nights run, rows written, share of unknown per field, reviewer corrections, cost. The inputs are published abstracts and the rows already carry their release tag, so a separate tracing tool would add little.
• The alternative is Langfuse, an open-source tool that records every model call with its input, output, cost and prompt version, and lets you replay one. It's the better pick when the pipeline has several steps, or when nothing downstream keeps the output, as in the company example below.

Build it: the regression set (60 min). Karim and Leonie grow the set from 15 to 38 cases. The new ones come from the corrections and the October inputs: four protocols, three conference abstracts, two corrections, two German abstracts, and the recurring failures. Every case notes where it came from, such as correction, 14 Oct, group B. Nine cases form the critical group: every protocol, the per-group sample size, the abstract with an instruction hidden in it, and the correction notices, because a retracted finding in a review is the worst outcome. Each critical case gets one extra line in the case file, group: critical under its metadata, so Promptfoo can run them on their own.

Before running anything they write the pass rule into the README, and the group leader signs it:

Pass rule (from 4 Nov 2026):
- every critical case passes in 3 of 3 repeats;
- at least 95% of all runs pass;
- no case that passes on the current release fails on the candidate.

The candidate is the proposed new version. In the workflow the rule becomes two test steps:

npx promptfoo@0.123.1 eval --repeat 3 --filter-metadata group=critical
PROMPTFOO_PASS_RATE_THRESHOLD=95 npx promptfoo@0.123.1 eval --repeat 3

The first runs only the critical cases, and a single failure turns the check red. The second runs all cases and fails only below 95%. The third line of the rule needs a person: Karim compares the candidate's results with the current release's, case by case, in the pull request.

The current release, extract-v3, fails all four protocol cases. That's the incident, now reproduced in the tests.

Build it: the fix through the gate. Leonie adds one rule to the prompt: "If the abstract describes a planned study and reports no results, set n to unknown and type to protocol." The pull request's test run is green on all 38 cases. The group leader approves, staging (the test copy of the pipeline) runs for two nights, and extract-v4 is released. Karim runs a query for all rows since 1 October whose type the new release marks as protocol, finds 32, and sends each group a list with the correct values. Two reviewers had already screened some into their reviews and correct them. He writes a half-page incident note: what happened, how it was found, what was affected, who was told, which case now guards against it.

Build it: a model swap that fails the gate. In November a fourth group asks to use the pipeline on unpublished clinical study reports. The data protection officer says those may not go to an external provider, but the institute's own model server, which IT runs with an open-weight model (one whose files anyone may download and run on their own machines), is approved. So Karim runs the same 38 cases against the institute's model, changing nothing else. Overall 94% pass. Two critical cases fail in every repeat: the hidden instruction and the per-group sample size. By the rule written in advance, the swap is blocked. He records the decision in the release log, the list of releases in the README: the model, the date, the failing cases, who decided (the group leader), and what happens next. The published-abstract pipeline stays on the current model. The clinical group's use waits until a prompt change for the institute's model passes the same gate, as a separate release.

Build it: the weekly run and the signals (30 min). The Monday workflow starts on a schedule:

on:
  schedule:
    - cron: '0 7 * * 1'

The cron line means minute 0, hour 7, any day of the month, any month, on Mondays. The workflow fetches the files of the production release's tag, runs the two test steps on them, then one query per signal. The shape query, written for a Postgres database, is short:

SELECT date_trunc('week', created_at) AS week,
       avg(CASE WHEN sample_size = 'unknown' THEN 1 ELSE 0 END) AS share_unknown
FROM extracted_rows GROUP BY week ORDER BY week;

It gives the share of unknown sample sizes for each week, the number that fell from about 20% to 9% in October. The last step sends the results to the chat channel's incoming webhook, a web address the chat tool gives you for posting messages from a program. A second, tiny job runs every morning at 7: it checks that the nightly job logged a finished run and posts "no run" when it didn't. That is the alert for a missing run.

Build it: handover. The runbook has five short sections: the weekly check, a signal looks wrong, release a change, roll back, a model or provider change. Each is five to ten steps, with the command and one sentence on what it does.

Then Sofia runs it without Karim touching the keyboard. She reruns the blocked swap from the release log and gets the same two failures. She rolls staging back from extract-v4 to extract-v3 and forward again. She reads the Monday post and explains what each number means. She gets stuck twice: she has no login for the institute's model server, and the runbook doesn't say who to call when the nightly run is missing. IT gives her an account; the runbook gains a line with IT's service desk number. The second time she does it alone in 25 minutes.

Last, the group leader asks the auditor's question a journal might ask one day: "Which version of the prompt extracted the sample size in row 4,812, and how did you check it?" Sofia answers from the folder in five minutes: the row's release tag, that release's test results, the week's signals and the reviewer who screened it. For how long to keep these records, and how to archive them with a paper, see T15-L05.

What it taught him: the tests had been right about everything they asked. The problem was everything they didn't ask, and the numbers that could have told him were already in the database.

Company: the sorting service and a model swap

Structure it. Tobias, the helpdesk administrator, operates the sorting service. Since the spring it's a small program that fetches each new email from the helpdesk, asks the model for a queue and writes the queue back; the email text stays in the helpdesk. Purchasing wants it moved to a model under the company's Microsoft Azure contract. "The sorting must stay as right as it is today through the switch, and afterwards, and my deputy Jana must be able to run it when I'm away. A good result: we switch only if the tests say so, and we'd notice a problem within a day."

Understand it. His set has 22 cases from T03-L03 and T03-L04. He adds what the last six months taught: twelve anonymised emails from past misroutes, eight in German, Dutch or French, because a third of the mail is. That makes 42, with ten critical: every cancellation, every account lockout and the hidden-instruction case. With the data protection officer he checks what the service records per email: ticket number, time, release, model, queue, cost. No text. To look at a failure, you open the ticket in the helpdesk, where its access rules and deletion rules already apply.

He checks the queue shares over the last three months: billing 31%, delivery 24%, technical 18%, returns 14%, cancellations 6%, other 7%. Those become his expected shape. The data protection officer asks one more thing: the weekly numbers show queues, not how fast each staff member works. That keeps the monitoring about the AI, and the works council, the elected staff representatives who have a say in anything that could watch employees, agrees without a formal agreement.

Choose tools. Promptfoo for the gate, on GitHub Actions, weekly and on every pull request. Langfuse, run by IT on a company server, for the per-email records and the weekly queue shares, because nothing else keeps a record of which model sorted which email. It's free to run yourself, but automatic deletion of old records needs a paid licence, so IT adds a monthly job that deletes records older than 90 days. The alternative, a table in the company's own database, would work for the records but has no ready-made charts or filters.

Build it. The candidate model passes 97% of runs overall. One critical case fails in two of three repeats: a Dutch cancellation phrased as a question, which it sorts into billing. By the rule, blocked. Tobias adds two sentences about cancellations phrased as questions to the prompt, as its own pull request. On the current model, all 42 still pass; on the candidate, all 42 pass. The switch is released as sort-v9, a new model and the new prompt together, labelled as such in the release log.

On the third day the weekly shape check, which Tobias now runs daily for two weeks after any switch, shows cancellations at 3% instead of the usual 6%. He rolls back to sort-v8 and the old model: six minutes, by the runbook. In Langfuse he filters the three days of sort-v9 for emails put in billing and exports their ticket numbers. A short script fetches those emails from the helpdesk and sorts them again with sort-v8: 19 come out as cancellations. The team moves them and replies to each customer that day. The cause is new: Swiss customers who write Kündigung, German for cancellation, inside a longer billing question. Five of them go into the set, anonymised, three as critical. The next candidate passes, and this time the shape holds.

Jana's drill follows the same pattern as Sofia's: rerun the blocked decision, roll back on staging, explain the weekly numbers. She gets stuck once, on where the Azure key lives, and the runbook gains the line.

In both cases the gate did its job twice: it blocked a change that failed a critical case, and it let through a fix only when every case passed. What caught the rest was a number somebody read every week.

What goes wrong

The set stops growing. Symptom: every release passes, while reviewers or customers keep reporting the same kinds of mistake. Fix: every confirmed failure gets one of three answers within a week: a new case, an existing case that already covers it, or a written decision to accept it, with a name and a date.

The average hides the critical failure. Symptom: 97% overall, and the one failing case is a cancellation. Fix: a critical group that must pass every time, written into the pass rule before the run.

The rule moves after the result. Symptom: a blocked swap is released because it's "close enough" and purchasing is waiting. Fix: the service owner signs the pass rule in advance. Any exception is a written decision with a name, a reason and an end date, never an edit to the rule.

Silence counts as a pass. Symptom: the weekly test run or the nightly job hasn't run for three weeks, and nobody noticed because nothing failed. Fix: alert on a missing run, not only on a failed one. The Monday post should say "no run" in words.

Records keep everything forever. Symptom: a tracing tool holds every customer email since launch, readable by everyone in IT. Fix: record what you need to find and replay a case, keep the content where it already lives, restrict who can read, and delete on a schedule someone checks.

Two things change in one release. Symptom: a new model and a new prompt go live together, and a problem can't be pinned on either. Fix: test them separately where you can. If they must go together, say so in the release log and watch more closely for two weeks.

The operation leaves with its builder. Symptom: only you have the key, understand the numbers or know how to roll back. Fix: a runbook, a substitute who has run it, and accounts in their name, done while you can still answer questions.

Do it yourself (4 hours, over a month)

1. Structure it (15 min). Write one sentence: what must stay right, for whom, through which changes, and who takes over. Ask the service owner to name your substitute.
2. Understand it (60 min). Compare what your inputs look like today with what your test set covers. Collect every correction or complaint since launch and sort them by cause. Chart one shape signal over the past months and find its usual range.
3. Choose tools (15 min). Decide where the weekly test run, the three signals and the records live. Use what your pipeline already writes before adding a tool. Check with your data protection officer what the records may contain and how long they're kept.
4. Build it: set and gate (70 min). Grow the set to 30 to 50 cases from real inputs and failures, each with its source. Mark the critical group. Write the pass rule and have the service owner sign it. Schedule the weekly run with an alert for a missing run.
5. Build it: signals and runbook (40 min). Set up the weekly post with the three signals and who responds to each. Write the runbook's five sections.
6. Build it: handover (40 min). Have your substitute rerun one gate decision and one rollback from the runbook while you watch without helping. Fix every place they got stuck. Answer one auditor question from the folder, with your substitute doing the answering.

Exit check

Delivered artifact: an operations folder for one production prompt, in its repository, containing:

• the regression set of 30 to 50 cases, each with its source, and the critical group marked;
• the pass rule, dated and signed before the first run it decided;
• the gate results for one change to the prompt, model or provider, and the decision;
• the monitoring page, a page in the repository listing the three signals, their usual ranges and who responds to each;
• one incident note or drill note, with the case it added;
• the runbook and your substitute's name.

It passes when the substitute reproduces the gate decision and a rollback from the runbook alone, and a colleague answers three questions from the folder: which release made a given output, how that release was tested, and what happened the last time something went wrong.

Rule to remember

A prompt stays right only while someone keeps testing it against what actually arrives, and that someone can't only be you.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T03-L04 · Release Prompt Changes Through Review, Tests and a Way Back: the repository, gate, tags and rollback this book operates.
• T15-L05: archiving AI-assisted results and handing research AI over when you leave.
• T05-L05: operating a whole set of automations, not one prompt.
• T11-L05: running your own model server, where the same gate decides every model change.
• T12-L05: governance records across all the AI systems your organisation runs.

Tools

• Promptfoo: Taught. The gate on every change and the weekly run.
• Langfuse: Taught. Records every model call with its release, cost and output, for monitoring and replay.
• Gitea: Taught. Scheduled workflows and the runbook's home in the lab example.
• GitHub: Catalogued. The same in the company example.
• Azure OpenAI: Catalogued. The candidate provider in the company's model swap.
• Claude Sonnet 5: Catalogued. The model both pipelines run on.