Last verified 23 September 2026. Promptfoo is free and open source; this book pins version 0.123.1. Gitea's review rules cost nothing. On GitHub, a private repository needs a paid plan such as Team for them. Details for experts at the end of this book.

The idea in one page

At Level 3 the test folder answered one question: is this change better than the last one? At Level 4 two more questions matter. Which version is running right now? And how fast can we go back? Five parts answer all three.

Two places, not two copies. Most teams need a staging setup, a copy of the pipeline that runs the new version on test data, and production, the one that writes to the real database or inbox. Both read the prompt from the same repository. Staging reads the newest accepted version. Production reads the tag you last released. Moving production to a new version is one reviewed line: the tag name.

The Level 4 questions. Because a prompt change now reaches real records, four questions come with it:

• Least privilege, the smallest access that does the job. Who can accept a change, who can release it, and what can the machines do? Two people can merge, one person releases, the test run can only read the repository, and its key can only spend a small test budget.
• Failure. What happens when a bad version gets through? You roll back, then find and fix what it wrote.
• Sign-off. IT runs the code server and holds the keys. The data protection officer decides whether test cases built from real emails or abstracts may live in the repository.
• Undo. The previous tag, a rehearsed rollback, and the tag in every output row.

The prompt itself is never your security. Keep the database permissions, the key handling and the human check outside it. A prompt says what you'd like; permissions decide what can happen.

Worked example: from a test folder to a release process

Both versions follow the four quadrants from T01-L01 · From Idea to Working Result: The Four-Quadrant Method. Follow the one closer to your work. The people, cases and numbers are examples written for this book.

Lab: the nightly extraction pipeline

Structure it (20 min). Leonie maintains the extraction prompt from T03-L03 · Put a Test Set Under the Prompt Your Team Depends On. Since the spring, Karim, the group's data manager, runs it every night on a small server of the institute. His script reads new abstracts from the literature feed, calls the Claude API and writes rows into the review database. Leonie writes: "Every change to the extraction prompt should pass the test set and one reviewer before it reaches the review database, and for every row we should be able to say which version made it. A good result: Karim can roll back in ten minutes without me."

Understand it (45 min). She starts where the prompt runs. On the server she finds the prompt file in Karim's script folder, and it differs from her folder's version 2 by one line: "If the outcome is unclear, choose the most likely one." A colleague added it on Friday, over a shared login. That line is why outcomes and sample sizes are guessed again. It had never met the test set.

She lists everyone who can change the prompt today. It's five people, through the shared server login, and nobody can say who did what. She also looks at what the rows in the review database record: the date, the abstract, the fields. Not the prompt version.

Then she asks IT about the institute's Gitea. IT says yes, with three conditions. The repository must be private to the group. Test cases must not contain unpublished data or personal data without the data protection officer's agreement. And any automatic test run uses the shared runner, the machine IT provides that carries out automatic jobs, so the API key must be stored as a secret, a value the code server hides from logs and from people reading the repository.

This changes the problem. She had thought of it as "put my folder in Git". The real gap is that nobody knows which version produced which row, and anyone with the server login can change it. Back to Structure it: "One source in Gitea. Only a tagged release reaches the server. Every row records its tag. The server can read the repository but can't change it."

Choose tools (15 min).

• The institute's Gitea, because IT runs it in the building, the abstracts and cases never leave the institute, and its branch protection costs nothing. The alternative, GitHub, would work as well, but the cases would sit with a US service, and IT would have to agree to that first.
• Gitea Actions to run Promptfoo on every pull request. Its workflow files mostly work on GitHub too, if the group ever moves.
• A separate test account area in the Claude Console, Anthropic's web page for managing API keys and spending. Anthropic calls these areas workspaces. Each has its own keys and its own spending limit, so a test key can't spend the production budget, and a leaked test key can't do much harm.
• Gitea releases for the named versions: a tag with notes, no special tool.

Build it: the repository and its rules (60 min). She creates the repository review-extraction on Gitea, private, with Karim and the group leader as the only other members. Into it go the prompt file, the contract, the cases, the Promptfoo configuration, the README and the saved results from T03-L03 · Put a Test Set Under the Prompt Your Team Depends On. The 12 real abstracts in the cases are already published, so they're fine. The data protection officer confirms by email that published abstracts need no further step.

Then she protects the main branch, as in T19-L03 · Propose, Review and Merge Changes With Pull Requests: no direct pushes, and one required approval. The automatic test run becomes a required check after its first run, as in T19-L04 · Check Every Change Automatically, Release Versions and Tell the Lab. A merge allowlist names who may merge: only she and Karim. The group leader is the reviewer for anything that changes what counts as a right answer.

Build it: the automatic test run (40 min). In the Claude Console she creates a workspace review-tests with a spending limit of 20 euros a month and one key. IT stores that key as the repository secret ANTHROPIC_TEST_KEY. Gitea switches Actions off for each new repository, so on the main settings page she ticks Enable Repository Actions. Then she adds the workflow file:

In plain words: on every pull request, fetch the files and set up Node.js 24, the program Promptfoo needs to run. Then npx downloads Promptfoo at exactly version 0.123.1 and runs the test set three times with fresh answers, saving the results in a file. The line runs-on names the kind of machine the job needs; she asks IT which name their runner answers to and uses that. Promptfoo ends with an error when any case fails, so a single failure turns the check red, and the branch rule blocks the merge. That matches the pass rule she wrote at Level 3, once she'd fixed the one wrong known answer: every case, every repeat.

To see the gate work, she proposes a small change in the browser, the way everyone will from now on: she edits the prompt file on Gitea and picks Create a new branch for this commit and start a pull request. A minute later the test run appears on the pull request. On GitHub the same button is called Propose changes.

Build it: releases the server can read but not change (45 min). Karim changes his script in three ways, and Leonie reviews each line.

First, the server gets its own read-only access to the repository, a deploy key: a key that lets one machine read one repository and nothing else. Karim creates an SSH key pair on the server, the same kind he uses to log in to servers, and pastes its public half into the repository under Settings, then Deploy Keys, leaving write access off. He then copies the repository to the server once with git clone. The shared login loses its copy of the prompt.

Second, a one-line settings file on the server names the release:

Every night, before it runs, the script fetches the tags from Gitea and reads the prompt file as it was in that release:

The first line brings the server's copy up to date; the second prints the prompt file from the named release, and the script passes that text to the model. Changing the settings line is the only way to change what production runs, and only Karim and Leonie can log in to the server.

Third, the script writes the tag into a new column, prompt_release, on every row.

Staging is a second copy of the script on the same server. It runs the same two lines with origin/main, the server's view of the main branch, in place of the release name, so it always reads the newest accepted version. It writes into a test table in the review database, never into the real one.

Build it: the first real change. The colleague who added Friday's line now proposes it properly, as a pull request. The test run goes red: two cases fail, the per-group sample size and the abstract with two outcomes, in all three repeats. The group leader reads the failures in the pull request and writes: "This is exactly the guessing the contract forbids." The change is closed, not merged. The colleague's real need, catching outcomes the prompt missed, goes into a new case instead.

A week later Leonie proposes a change that passes: a clearer rule for secondary outcomes, with two new cases. The test run is green, the group leader approves, Leonie merges. Staging runs it for two nights on the test table, and she checks 20 rows by hand. Then she publishes the release in Gitea: Releases, New Release, tag extract-v3 on main, with notes that say what changed and who reviewed it. Karim changes the release line on the server from extract-v2 to extract-v3 and notes the date in the README's release list.

Build it: rehearse the way back (20 min). Before anyone needs it, Karim rehearses the rollback on staging from the README alone. He sets the release line back to extract-v2, runs the script once by hand, and checks that new rows say extract-v2. Then he runs a query that lists every row written by extract-v3, because that's how the group would find a bad release's output:

It picks out exactly the rows whose release column holds that tag. Seven minutes in all. He adds one missing line to the README, the command that runs the script by hand.

Where it stops. The group asks whether the test set should also run every week against the live model, to catch the model itself changing. Watching a prompt in production over months, with a growing set of cases from real failures, is Level 5 (T03-L05 · Keep a Production Prompt Healthy for Months, and Hand It Over).

What it taught her: the test set was never the weak point. The weak point was that the prompt the tests checked wasn't the prompt that ran.

Company: the helpdesk sorting prompt

Structure it (20 min). Aylin runs customer support at the lab-supplies distributor from T03-L03 · Put a Test Set Under the Prompt Your Team Depends On. Her tested sorting prompt, version 4, runs inside the helpdesk system's AI feature. "Every change to the sorting prompt should pass the test set and the support manager before customers feel it, and we should be able to go back to the previous version in minutes."

Understand it (45 min). She asks the helpdesk administrator, Tobias, how the prompt gets into the helpdesk. The answer changes her plan. The helpdesk's AI feature takes the prompt as text in an admin screen. It can't read from a repository, and there's no way to load it automatically. And four people have admin rights, because admin rights also cover unrelated settings such as holiday replies.

She checks the test cases with the data protection officer. They were built from real emails with names, addresses and order numbers replaced by made-up ones. The data protection officer agrees they may live in the company's GitHub repository, which is covered by the company's contract with GitHub, on one condition: nothing goes in that could identify a customer, including rare product combinations that point to one buyer. Aylin removes two such cases and writes two made-up ones instead.

Back to Structure it: "The repository is the source; the helpdesk is a copy. The copy must say which release it is, only one person may change it, and going back must be a practised two-minute paste."

Choose tools. The company's GitHub, on the Team plan it already pays for, because the product team keeps its code there and branch protection is included. GitHub Actions, its built-in automation, runs the same Promptfoo workflow as Leonie's, stored in .github/workflows/. The test key comes from its own Claude Console workspace with a monthly limit of 20 euros.

Build it. The repository helpdesk-sorting has the same parts and the same branch rules: one approval, from the support manager, and a green test run before merging. The first line of the prompt file is now a comment: # release sort-v5. The sorting model ignores that line, and anyone looking at the admin screen can see which release is live.

Tobias asks IT to split the admin role. Only he and his deputy can edit the AI prompt; the team leads keep holiday replies and templates. The release steps go into the README: merge, tag, open the tag's prompt file on GitHub (pick the tag in the branch menu above the file list, open the file, and use the copy button), paste it into the admin screen, save, send the three test emails from the README to the test mailbox, and check their queues. Rollback is the same with the previous tag.

The first release, sort-v5, adds the urgency field from T03-L03 · Put a Test Set Under the Prompt Your Team Depends On. Three weeks later a product manager asks for a new warranty queue. The pull request's test run is green, the support manager approves, and Tobias releases sort-v6. Next morning, six returns emails land in warranty. The test set hadn't included returns that mention a defect. Tobias pastes sort-v5 back in: four minutes. Aylin runs a helpdesk search for emails sorted between the two releases, and the team moves the six emails by hand. Then she adds the failing emails, anonymised, as cases, and the fixed version goes through the same gate as sort-v7.

In both cases the change that went wrong was small and well meant. What saved the day wasn't a better prompt. It was knowing which version ran, and having a way back that someone had practised.

What goes wrong

The admin screen is the real source. Symptom: the text in production differs from the repository, and nobody can say when it changed. Fix: production reads only a tag, or shows its release name in the first line. Restrict who can edit it to the person who releases and a deputy.

Everyone can merge. Symptom: the author approves their own change on a Friday evening. Fix: protect the main branch. Require one approval from someone other than the author and a green test run. Limit merging to two people.

The test key is the production key. Symptom: a stuck test run spends the month's budget, or a leaked key reaches the live system's account. Fix: a separate workspace, its own key stored as a secret, and a low spending limit. The production key never leaves the production server.

Real data in the test cases. Symptom: customer emails or unpublished results sit in a repository that more people can read than the helpdesk or the lab notebook. Fix: anonymise or write cases by hand, and get the data protection officer's agreement before the first commit. If real data got in, tell IT: deleting the file doesn't remove it from the history.

Rollback fixes tomorrow, not yesterday. Symptom: the old prompt is back, and the wrong rows or misrouted emails are still there. Fix: record the release in every output, then search by release and correct what the bad one produced. Tell the people who worked from those outputs.

Everything changes in one release. Symptom: a new prompt, a new model and a new case file go live together, and nobody can say what caused the failure. Fix: one change per pull request. If two must move together, say so in the pull request and test them together.

A rollback nobody has practised. Symptom: the only person who knows the steps is on holiday. Fix: rehearse it once on staging, from the README alone, done by someone who didn't write it.

Do it yourself (4 hours, over one or two weeks)

1. Structure it (20 min). Write one sentence: which prompt, which real system it writes to, who must approve changes, and how fast you need to go back.
2. Understand it (45 min). Find every place the prompt runs and every copy of it. List who can change each copy today. Check what outputs record about their prompt version. Ask IT which code server to use and on what conditions, and ask the data protection officer about your test cases.
3. Choose tools (15 min). Pick the code server IT approves. Create a separate test workspace with its own key and spending limit, and have IT store the key as a secret. Decide how production will read a tag, or, if it can't, how the release name will show.
4. Build it: repository and gate (100 min). Create the private repository and add the test folder. Protect the main branch: one approval, a green test run, two people who may merge. Switch on Actions for the repository, add the workflow file, and open a test pull request that should fail, to see the check turn red. Then tick the test run as a required check in the branch rule.
5. Build it: release and way back (60 min). Give production a read-only deploy key and a settings line naming the release, make it read the prompt from that tag, and record the tag in every output. Release one real change through the gate. Have a colleague rehearse the rollback on staging from the README, time it, and fix the README where they got stuck. Send IT the access list to confirm.

Exit check

What you hand in: a prompt repository with one released change, containing:

• the pull request with its automatic test run and the reviewer's approval;
• the branch rules: no direct changes, one approval, a green test run, and who may merge;
• the release tag and a sample of production outputs that record it;
• a README with the release steps, the rollback steps and the release list;
• IT's written confirmation of who has access to the repository, the server or admin screen, and the keys.

It passes when a colleague rolls production, or staging if production can't be touched, back to the previous tag from the README alone in under ten minutes, and can list the outputs the newer release produced.

Rule to remember

If you can't say which version made a result, and go back to the one before, it isn't released.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T03-L03 · Put a Test Set Under the Prompt Your Team Depends On: the test folder this book puts under version control.
• T03-L05 · Keep a Production Prompt Healthy for Months, and Hand It Over: keeping a prompt healthy in production over months, and handing it over.
• T19-L03 · Propose, Review and Merge Changes With Pull Requests and T19-L04 · Check Every Change Automatically, Release Versions and Tell the Lab: the pull requests, protected branches, automatic checks and releases this book uses.
• T05-L04 · Make an Automation Safe to Fail Before It Touches Real Systems: making the automation around the prompt safe to fail.
• T12-L04 · Lock Down a Self-Hosted AI Service Before Real Data Goes In: securing the servers and keys a prompt pipeline depends on.

Tools

• Git: Taught. The history of every change, and the tags you release.
• Gitea: Taught. The code server IT runs in the building, with branch protection and Gitea Actions.
• GitHub: Taught. The company's code server in the second example, with GitHub Actions.
• Promptfoo: Taught. The test runner that decides whether a change may be merged.
• Claude API: Catalogued. The model access the pipeline and the tests call.
• Langfuse: Catalogued. Stores named prompt versions and records model calls, if you later want release labels and records outside Git.