Heidelberg AI Curriculum

T03-L04 · Prompting & context · Level 4 Integrator · 18-minute read, plus 4 hours of doing

Two copies of the same prompt

The clinical research group's extraction prompt has grown up. At Level 3 a PhD student ran it by hand with a test set in a folder. Now it runs every night on the institute's server, reads new abstracts from the literature feed and writes rows straight into the group's review database, where four people screen them. On Monday a reviewer notices that sample sizes are back to being guessed. The test folder says the prompt is fine. It is: the folder holds version 2. The server holds a copy someone edited on Friday "to catch more outcomes". Nobody knows which of the 340 rows written since then came from which copy.

In a company the prompt sorts about 200 customer emails a day into queues inside the helpdesk system. Its text lives in the helpdesk's admin screen. A team lead with admin rights pasted in a new version on Thursday afternoon, and by Friday morning cancellation requests were sitting in the general queue. The old text is gone.

You are at Level 4 Integrator. The prompt now touches real systems: a review database, a customer inbox. A bad change reaches real records and real customers before anyone reads them. This book shows you how to make every change pass tests and a person, release it under a name, and undo it in minutes.

After this you can

Move a shared prompt and its test set into a repository on your organisation's code server, with access limited to the people who need it.
Make every prompt change go through a pull request that runs the test set automatically and needs one named reviewer's approval.
Release a tested version to production by its tag, and record that tag with every output the prompt produces.
Roll production back to the previous release in under ten minutes, and find the outputs the bad release produced.
Get IT and your data protection officer to sign off where the repository, the test cases and the keys live.

Before you start

Books: T01-L01 for the four-quadrant method. T03-L03 for the test folder this book builds on: a contract (the written rules for what counts as a right answer), at least 15 cases with known answers, a Promptfoo configuration and a README. T08-L02 if you've never used Git.
Git, the free program that keeps a history of every change to a set of files, installed on your computer.
A code server your organisation runs or pays for, where the history is kept and reviewed. In a lab that is usually the institute's Gitea, free software IT runs on its own machines. In a company it is often GitHub on a paid plan. Ask IT which one you may use for this data.
Promptfoo, the test runner from T03-L03, and your test folder.
An API key for tests only, from a separate account area with its own spending limit. The worked example shows how.
A colleague as reviewer: someone who knows what a right answer looks like, such as the group leader or the support manager.
About 4 hours 20 minutes: 18 to read, about 4 hours to do, over one or two weeks, plus waiting for IT.

Last verified 23 September 2026. Promptfoo is free and open source; this book pins version 0.123.1. Gitea's review rules cost nothing. On GitHub, a private repository needs a paid plan such as Team for them. Details for experts at the end of this book.

The idea in one page

At Level 3 the test folder answered one question: is this change better than the last one? At Level 4 two more questions matter. Which version is running right now? And how fast can we go back? Five parts answer all three.

Part	What it is	What it prevents
One source	The prompt, its tests and its README in one repository, a folder whose full history Git keeps on the code server.	Copies that drift apart on laptops, servers and admin screens.
A gate	Every change arrives as a pull request, a proposed change that others can read and comment on before it's accepted. The test set runs on it automatically, and a named reviewer approves it. Only then may someone merge it: accept it into the main branch, the line of history that counts as accepted.	A change nobody tested or read.
A named release	Production runs one exact version, marked with a tag, a fixed name such as `extract-v3` attached to one point in the history.	"Which prompt made this row?" with no answer.
A record	Every output carries the tag of the prompt that made it.	A bad week you can't find in the database.
A way back	Rolling back means pointing production at the previous tag, practised once before you need it.	Rewriting the old text from memory.

Two places, not two copies. Most teams need a staging setup, a copy of the pipeline that runs the new version on test data, and production, the one that writes to the real database or inbox. Both read the prompt from the same repository. Staging reads the newest accepted version. Production reads the tag you last released. Moving production to a new version is one reviewed line: the tag name.

The Level 4 questions. Because a prompt change now reaches real records, four questions come with it:

Least privilege, the smallest access that does the job. Who can accept a change, who can release it, and what can the machines do? Two people can merge, one person releases, the test run can only read the repository, and its key can only spend a small test budget.
Failure. What happens when a bad version gets through? You roll back, then find and fix what it wrote.
Sign-off. IT runs the code server and holds the keys. The data protection officer decides whether test cases built from real emails or abstracts may live in the repository.
Undo. The previous tag, a rehearsed rollback, and the tag in every output row.

The prompt itself is never your security. Keep the database permissions, the key handling and the human check outside it. A prompt says what you'd like; permissions decide what can happen.

Worked example: from a test folder to a release process

Both versions follow the four quadrants from T01-L01. Follow the one closer to your work. The people, cases and numbers are examples written for this book.

Lab: the nightly extraction pipeline

Structure it (20 min). Leonie maintains the extraction prompt from T03-L03. Since the spring, Karim, the group's data manager, runs it every night on a small server of the institute. His script reads new abstracts from the literature feed, calls the Claude API and writes rows into the review database. Leonie writes: "Every change to the extraction prompt should pass the test set and one reviewer before it reaches the review database, and for every row we should be able to say which version made it. A good result: Karim can roll back in ten minutes without me."

Understand it (45 min). She starts where the prompt runs. On the server she finds the prompt file in Karim's script folder, and it differs from her folder's version 2 by one line: "If the outcome is unclear, choose the most likely one." A colleague added it on Friday, over a shared login. That line is why outcomes and sample sizes are guessed again. It had never met the test set.

She lists everyone who can change the prompt today. It's five people, through the shared server login, and nobody can say who did what. She also looks at what the rows in the review database record: the date, the abstract, the fields. Not the prompt version.

Then she asks IT about the institute's Gitea. IT says yes, with three conditions. The repository must be private to the group. Test cases must not contain unpublished data or personal data without the data protection officer's agreement. And any automatic test run uses the shared runner, the machine IT provides that carries out automatic jobs, so the API key must be stored as a secret, a value the code server hides from logs and from people reading the repository.

This changes the problem. She had thought of it as "put my folder in Git". The real gap is that nobody knows which version produced which row, and anyone with the server login can change it. Back to Structure it: "One source in Gitea. Only a tagged release reaches the server. Every row records its tag. The server can read the repository but can't change it."

Choose tools (15 min).

The institute's Gitea, because IT runs it in the building, the abstracts and cases never leave the institute, and its branch protection costs nothing. The alternative, GitHub, would work as well, but the cases would sit with a US service, and IT would have to agree to that first.
Gitea Actions, Gitea's built-in automation, to run Promptfoo on every pull request. Its instructions live in short text files that are mostly compatible with GitHub's, so the steps carry over if the group ever moves.
A separate test account area in the Claude Console, Anthropic's web page for managing API keys and spending. Anthropic calls these areas workspaces. Each has its own keys and its own spending limit, so a test key can't spend the production budget, and a leaked test key can't do much harm.
Tags for releases, not a special tool: Git has them built in.

Build it: the repository and its rules (60 min). She creates the repository review-extraction on Gitea, private, with Karim and the group leader as the only other members. Into it go the prompt file, the contract, the cases, the Promptfoo configuration, the README and the saved results from T03-L03. The 12 real abstracts in the cases are already published, so they're fine. The data protection officer confirms by email that published abstracts need no further step.

Then she protects the main branch. In the repository she opens Settings, then Branches, and adds a rule for main with three parts. First, Disable push: nobody may send changes from their computer straight into the main branch, so every change has to come as a pull request. Second, one required approval. Third, the automatic test run must pass. Gitea lists the test runs it has seen in the last week, so she ticks this third part after the first test run below. A merge allowlist names who may merge: only she and Karim. The group leader is the reviewer for anything that changes what counts as a right answer.

Build it: the automatic test run (40 min). In the Claude Console she creates a workspace review-tests with a spending limit of 20 euros a month and one key. IT stores that key as a repository secret named ANTHROPIC_TEST_KEY, under Settings, then Actions, then Secrets. Gitea switches Actions off for each new repository, so on the main settings page she ticks Enable Repository Actions. Then she adds a workflow file, a short file that tells Gitea what to run and when:

# .gitea/workflows/prompt-tests.yaml
name: prompt-tests
on: pull_request
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 24
      - run: npx promptfoo@0.123.1 eval --no-cache --repeat 3 -o results.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_TEST_KEY }}

In plain words: on every pull request, fetch the files and set up Node.js 24, the program Promptfoo needs to run. Then npx downloads Promptfoo at exactly version 0.123.1 and runs the test set three times with fresh answers, saving the results in a file. The line runs-on names the kind of machine the job needs; she asks IT which name their runner answers to and uses that. Promptfoo ends with an error when any case fails, so a single failure turns the check red, and the branch rule blocks the merge. That matches the pass rule she wrote at Level 3, once she'd fixed the one wrong known answer: every case, every repeat.

To see the gate work, she proposes a small change the way everyone will from now on. On Gitea's web page she opens the prompt file, clicks the pencil to edit it, and at the bottom picks Create a new branch for this commit and start a pull request. A commit is one saved change in the history, and a branch is a side line of history where commits wait until they're merged. The pull request opens, and a minute later the test run appears on it. On GitHub the same button is called Propose changes.

Build it: releases the server can read but not change (45 min). Karim changes his script in three ways, and Leonie reviews each line.

First, the server gets its own read-only access to the repository, a deploy key: a key that lets one machine read one repository and nothing else. Karim creates an SSH key pair on the server, the same kind he uses to log in to servers, and pastes its public half into the repository under Settings, then Deploy Keys, leaving write access off. He then copies the repository to the server once with git clone. The shared login loses its copy of the prompt.

Second, a one-line settings file on the server names the release:

PROMPT_RELEASE=extract-v3

Every night, before it runs, the script fetches the tags from Gitea and reads the prompt file as it was in that release:

git -C /srv/review-extraction fetch --tags
git -C /srv/review-extraction show "$PROMPT_RELEASE:prompt.md"

The first line brings the server's copy up to date; the second prints the prompt file from the named release, and the script passes that text to the model. Changing the settings line is the only way to change what production runs, and only Karim and Leonie can log in to the server.

Third, the script writes the tag into a new column, prompt_release, on every row.

Staging is a second copy of the script on the same server. It runs the same two lines with origin/main, the server's view of the main branch, in place of the release name, so it always reads the newest accepted version. It writes into a test table in the review database, never into the real one.

Build it: the first real change. The colleague who added Friday's line now proposes it properly, as a pull request. The test run goes red: two cases fail, the per-group sample size and the abstract with two outcomes, in all three repeats. The group leader reads the failures in the pull request and writes: "This is exactly the guessing the contract forbids." The change is closed, not merged. The colleague's real need, catching outcomes the prompt missed, goes into a new case instead.

A week later Leonie proposes a change that passes: a clearer rule for secondary outcomes, with two new cases. The test run is green, the group leader approves, Leonie merges. Staging runs it for two nights on the test table, and she checks 20 rows by hand. Then she creates the release from her own copy of the repository:

git switch main
git pull
git tag extract-v3
git push origin extract-v3

The first two lines bring her copy of the main branch up to date with Gitea. The third puts the name extract-v3 on the version she just checked; the last sends that name to Gitea. Karim changes the release line on the server from extract-v2 to extract-v3 and notes the date in the README's release list.

Build it: rehearse the way back (20 min). Before anyone needs it, Karim rehearses the rollback on staging from the README alone. He sets the release line back to extract-v2, runs the script once by hand, and checks that new rows say extract-v2. Then he runs a query that lists every row written by extract-v3, because that's how the group would find a bad release's output:

SELECT * FROM extracted_rows WHERE prompt_release = 'extract-v3';

It picks out exactly the rows whose release column holds that tag. Seven minutes in all. He adds one missing line to the README, the command that runs the script by hand.

Where it stops. The group asks whether the test set should also run every week against the live model, to catch the model itself changing. Watching a prompt in production over months, with a growing set of cases from real failures, is Level 5 (T03-L05).

What it taught her: the test set was never the weak point. The weak point was that the prompt the tests checked wasn't the prompt that ran.

Company: the helpdesk sorting prompt

Structure it (20 min). Aylin runs customer support at the lab-supplies distributor from T03-L03. Her tested sorting prompt, version 4, runs inside the helpdesk system's AI feature. "Every change to the sorting prompt should pass the test set and the support manager before customers feel it, and we should be able to go back to the previous version in minutes."

Understand it (45 min). She asks the helpdesk administrator, Tobias, how the prompt gets into the helpdesk. The answer changes her plan. The helpdesk's AI feature takes the prompt as text in an admin screen. It can't read from a repository, and there's no way to load it automatically. And four people have admin rights, because admin rights also cover unrelated settings such as holiday replies.

She checks the test cases with the data protection officer. They were built from real emails with names, addresses and order numbers replaced by made-up ones. The data protection officer agrees they may live in the company's GitHub repository, which is covered by the company's contract with GitHub, on one condition: nothing goes in that could identify a customer, including rare product combinations that point to one buyer. Aylin removes two such cases and writes two made-up ones instead.

Back to Structure it: "The repository is the source; the helpdesk is a copy. The copy must say which release it is, only one person may change it, and going back must be a practised two-minute paste."

Choose tools. The company's GitHub, on the Team plan it already pays for, because the product team keeps its code there and branch protection is included. GitHub Actions, its built-in automation, runs the same Promptfoo workflow as Leonie's, stored in .github/workflows/. The test key comes from its own Claude Console workspace with a monthly limit of 20 euros.

Build it. The repository helpdesk-sorting has the same parts and the same branch rules: one approval, from the support manager, and a green test run before merging. The first line of the prompt file is now a comment: # release sort-v5. The sorting model ignores that line, and anyone looking at the admin screen can see which release is live.

Tobias asks IT to split the admin role. Only he and his deputy can edit the AI prompt; the team leads keep holiday replies and templates. The release steps go into the README: merge, tag, open the tag's prompt file on GitHub (pick the tag in the branch menu above the file list, open the file, and use the copy button), paste it into the admin screen, save, send the three test emails from the README to the test mailbox, and check their queues. Rollback is the same with the previous tag.

The first release, sort-v5, adds the urgency field from T03-L03. Three weeks later a product manager asks for a new warranty queue. The pull request's test run is green, the support manager approves, and Tobias releases sort-v6. Next morning, six returns emails land in warranty. The test set hadn't included returns that mention a defect. Tobias pastes sort-v5 back in: four minutes. Aylin runs a helpdesk search for emails sorted between the two releases, and the team moves the six emails by hand. Then she adds the failing emails, anonymised, as cases, and the fixed version goes through the same gate as sort-v7.

In both cases the change that went wrong was small and well meant. What saved the day wasn't a better prompt. It was knowing which version ran, and having a way back that someone had practised.

What goes wrong

The admin screen is the real source. Symptom: the text in production differs from the repository, and nobody can say when it changed. Fix: production reads only a tag, or shows its release name in the first line. Restrict who can edit it to the person who releases and a deputy.

Everyone can merge. Symptom: the author approves their own change on a Friday evening. Fix: protect the main branch. Require one approval from someone other than the author and a green test run. Limit merging to two people.

The test key is the production key. Symptom: a stuck test run spends the month's budget, or a leaked key reaches the live system's account. Fix: a separate workspace, its own key stored as a secret, and a low spending limit. The production key never leaves the production server.

Real data in the test cases. Symptom: customer emails or unpublished results sit in a repository that more people can read than the helpdesk or the lab notebook. Fix: anonymise or write cases by hand, and get the data protection officer's agreement before the first commit. If real data got in, tell IT: deleting the file doesn't remove it from the history.

Rollback fixes tomorrow, not yesterday. Symptom: the old prompt is back, and the wrong rows or misrouted emails are still there. Fix: record the release in every output, then search by release and correct what the bad one produced. Tell the people who worked from those outputs.

Everything changes in one release. Symptom: a new prompt, a new model and a new case file go live together, and nobody can say what caused the failure. Fix: one change per pull request. If two must move together, say so in the pull request and test them together.

A rollback nobody has practised. Symptom: the only person who knows the steps is on holiday. Fix: rehearse it once on staging, from the README alone, done by someone who didn't write it.

Do it yourself (4 hours, over one or two weeks)

Structure it (20 min). Write one sentence: which prompt, which real system it writes to, who must approve changes, and how fast you need to go back.
Understand it (45 min). Find every place the prompt runs and every copy of it. List who can change each copy today. Check what outputs record about their prompt version. Ask IT which code server to use and on what conditions, and ask the data protection officer about your test cases.
Choose tools (15 min). Pick the code server IT approves. Create a separate test workspace with its own key and spending limit, and have IT store the key as a secret. Decide how production will read a tag, or, if it can't, how the release name will show.
Build it: repository and gate (100 min). Create the private repository and add the test folder. Protect the main branch: one approval, a green test run, two people who may merge. Switch on Actions for the repository, add the workflow file, and open a test pull request that should fail, to see the check turn red. Then tick the test run as a required check in the branch rule.
Build it: release and way back (60 min). Give production a read-only deploy key and a settings line naming the release, make it read the prompt from that tag, and record the tag in every output. Release one real change through the gate. Have a colleague rehearse the rollback on staging from the README, time it, and fix the README where they got stuck. Send IT the access list to confirm.

Exit check

What you hand in: a prompt repository with one released change, containing:

the pull request with its automatic test run and the reviewer's approval;
the branch rules: no direct changes, one approval, a green test run, and who may merge;
the release tag and a sample of production outputs that record it;
a README with the release steps, the rollback steps and the release list;
IT's written confirmation of who has access to the repository, the server or admin screen, and the keys.

It passes when a colleague rolls production, or staging if production can't be touched, back to the previous tag from the README alone in under ten minutes, and can list the outputs the newer release produced.

Rule to remember

If you can't say which version made a result, and go back to the one before, it isn't released.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

What	Details	What it means for you
Promptfoo	Version 0.123.1, from 18 September 2026. `promptfoo eval` ends with exit code 100 when any case fails (the number a program reports when it stops; anything but 0 counts as failure). The variable `PROMPTFOO_PASS_RATE_THRESHOLD` sets a lower pass rate in percent; without it, every case must pass. Results go online only with `--share`.	The exit code is what turns the automatic check red. Leave the threshold unset if your pass rule is "every case".
Gitea Actions	Gitea's automation, available since Gitea 1.19 and switched on by default since 1.21. Workflow files go in `.gitea/workflows/`. Jobs run on a Gitea Runner, a separate program IT installs. Secrets can be set for a user, an organisation or a repository; the most specific one wins.	Ask IT which Gitea version runs and whether a runner is available for your repository.
Gitea branch protection	Options include required approvals, status checks (jobs that must pass), a merge allowlist (who may merge) and protected file patterns (files that no push or pull request may change).	This is where you set the three rules from the worked example.
GitHub	Protected branches and required reviewers in private repositories come with the Team plan, not the Free plan. Private repositories include 2,000 Actions minutes (running time on GitHub's machines) a month on Free and 3,000 on Team. Workflow files go in `.github/workflows/`.	A test run of 15 cases takes a few minutes, so the included minutes cover hundreds of runs.
Promptfoo on GitHub	`promptfoo/promptfoo-action@v1`, a ready-made workflow step, runs the tests on pull requests that change prompt files and comments with a before-and-after comparison.	Optional; the plain `npx` step shown in the book works on GitHub and Gitea alike.
Claude Console workspaces	API keys belong to the workspace they were created in and can't be moved. Each workspace can have its own spending limit, lower than the organisation's.	One workspace for tests, one for production.
Model	`claude-sonnet-5`, US$2 per million input tokens and US$10 per million output tokens (tokens are small pieces of text). Anthropic won't retire it before 30 June 2027.	Moving to a new model is a pull request like any other. Watching it over months: T03-L05.

Release Prompt Changes Through Review, Tests and a Way Back