Heidelberg AI Curriculum

T15-L02 · AI in research · Level 2 Power user · 13-minute read, plus 60 minutes of doing

March doesn't reproduce in September

In March a PhD student used an AI assistant to screen 180 abstracts for a systematic review against four inclusion criteria. She kept the final list, 41 papers, and wrote a line in her use log. In September a journal reviewer asks the authors to describe the screening so it could be repeated. Her co-author, who does the second screening, pastes what looks like the same prompt into the same chat app. Nine decisions come out differently. Nobody can say whether the prompt changed, the model behind the app changed, the chat's memory played a part, or the answers simply vary from run to run.

In a company it's the quality team of a medical-device supplier. Every quarter an assistant sorts about 400 customer complaints into eight defect categories for the management review. An auditor asks how complaint 2026-0417 ended up as packaging and not sterility. The colleague who ran the sorting has moved to another site, and the chat history is in her account.

At Level 1 you logged each AI use and wrote a disclosure (T15-L01). You are now at Level 2 Power user: the AI step feeds shared work, and a co-author, a reviewer or an auditor may ask for it again when you're not there. This book shows you how to record one AI step so that someone else can rerun it, and how to report honestly when the second run differs.

After this you can

Say why a chat window can't be rerun reliably, and choose a pinned model or a local model file for the step that goes into shared work.
Keep a run record: input and its checksum, prompt version, exact model, settings, date and the unedited raw output.
Decide before rerunning what counts as the same answer, and compare runs on that rule.
Run the step twice from the start, and send every disagreement to a human decision instead of picking the run you prefer.
Have a co-author rerun the step from the folder alone, and write a Methods sentence that states the agreement.

Before you start

Books: T01-L01 for the four-quadrant method. T15-L01 for the use log and disclosure this book extends. T10-L02 for keeping raw files untouched.
A way to call a pinned model. This book uses the Claude Console, where you pick the exact model ID and run a prompt in the browser, without code. It's the pick when your material may go to a cloud service. If the material must stay on your computer, or the step has to be rerun years from now, use a model file in LM Studio instead (see the company example). Use only an account and material your organisation allows (T12-L01).
One AI step from your own research or project, with input you're allowed to use and a result a human can check.
A shared project folder your co-author can open.
One co-author or colleague who will rerun the step.
About 75 minutes: 13 to read, about 60 to do.

Last verified 23 September 2026. Claude API: every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on. Claude Sonnet 5 has the ID claude-sonnet-5, costs $2 per million input tokens and $10 per million output tokens, and won't be retired before 30 June 2027. Models released after Claude Opus 4.6 don't accept a temperature setting other than 1.0, so you can't turn down their randomness. The Claude Console lets you write and test prompts in the browser. LM Studio lets you set the temperature and a seed; its documentation says a fixed seed makes the random sampling repeat across runs.

The idea in one page

A chat window is a moving target. The model behind the name in the app can change. The app adds its own instructions, may switch on search or memory, and doesn't show you all of it. That's fine for exploring. For a step that goes into a paper, a report or an audit trail, use a route where you can name the exact model and see everything that went in: an API model ID, or a model file on your own computer.

Same model isn't same answer. Current cloud models give slightly different answers to the same prompt, and you can't switch that off. So don't aim for the same words. Aim for the same decisions: for each item, the same include or exclude, the same category. Decide that rule before you rerun, and write it down. Then two runs can be compared, and a difference is a finding, not an embarrassment.

Three levels, and say which one you mean:

Level	Means	Realistic for
Same words	The output file is identical	A local model with a fixed seed and settings, on the same machine
Same decisions	Every item gets the same label	The goal for most research and quality steps
Same method	Someone else can run the recorded step, even if some answers differ	The minimum for anything that goes into shared work

The run record. One short text file next to the results. It answers every question a reviewer or auditor will ask:

Step:        what it does, why, who ran it
Input:       file name, checksum, where it's kept
Prompt:      file name and version (never edited in place)
Model:       exact model ID, or model file name and checksum
Software:    app or service, and its version if shown
Settings:    temperature, seed, anything switched off; "not settable" if so
Date:        of each run
Raw output:  one unedited file per run
Same means:  the comparison rule, written before the rerun
Result:      agreement, the list of differences, and who decided them

A checksum is a short code computed from a file's exact bytes. If one character in the file changes, the checksum changes. It tells your co-author they're rerunning the same input, not a similar one:

Windows (PowerShell):  Get-FileHash abstracts.txt
macOS or Linux:        shasum -a 256 abstracts.txt

Run twice, keep both. Run the step twice from the start before anyone relies on it. The items where the runs disagree are the ones the model finds hard, and they go to a human. Never rerun until you get the answer you like and keep only that one. Number every run and keep every raw output.

Worked example: two AI steps, four quadrants

The method comes from T01-L01. Here it is for a systematic review, then for a quality report.

Lab: screening abstracts for a systematic review

Structure it. Hanna, the PhD student, is redoing the screening properly after the reviewer's question. "180 PubMed abstracts, four inclusion criteria, an include, exclude or unclear decision for each. A useful result: a folder that lets Lukas, my second screener, rerun the AI step on his own laptop and compare decisions with mine, and a Methods sentence the reviewer can check."

Understand it. She reruns her March prompt twice in the chat app. The two runs disagree on 7 of 180 decisions, and both differ from March. She looks at the seven: four abstracts don't mention the age range, two describe a mixed population, and one is a protocol with no results. They're the borderline cases a human screener also hesitates over. She also finds that the temperature setting she planned to fix at zero doesn't exist for current Claude models. That changes her problem. She can't make the model repeat itself exactly. What she can make repeatable is everything around it, and she can measure the part that varies. Back to Structure it: "Aim for the same decisions, not the same words. Run twice, report the agreement, and give every disagreement to a human screener."

Choose tools. The Claude Console with Claude Sonnet 5, because its model ID names one fixed snapshot, the abstracts are published, and the console shows everything that goes into the request. The chat app stays for exploring. She considers a local model in LM Studio, which could repeat itself exactly. She rejects it for now: the small models her laptop runs misjudged the study designs more often in her test on ten abstracts. She notes the retirement date in the run record, because after it nobody can rerun this model.

Build it. Her project folder:

screening/
  input/abstracts-2026-09-02.txt        PubMed export, never edited
  prompt/screen-v2.txt                  v1 kept, never overwritten
  runs/run-01.txt  run-02.txt           raw output, unedited
  run-record.txt
  comparison.xlsx                       one row per PMID

The prompt asks for one line per PMID with include, exclude or unclear and the sentence from the abstract that decides it. Her rule, written in the record before run 2: "Same means the same decision for every PMID. Wording of the quoted sentence doesn't count." Run 1 and run 2 agree on 174 of 180. She and Lukas screen the six disagreements by hand, together with all 23 unclear decisions. The checksum of the input goes into the record, and so does the date of each run.

Then the test. Lukas opens the folder on his laptop, checks the input checksum, pastes screen-v2.txt into the console with the same model ID and runs it once. His run agrees with run 1 on 175 of 180. Four of his five differences are among the six they already screened by hand. The fifth is new, so they screen it too. Her Methods sentence:

Two authors used Claude Sonnet 5 (claude-sonnet-5, via the Claude Console) with a fixed prompt (supplementary file S2) to pre-screen 180 abstracts. Three independent runs agreed on 173 of 180 decisions. All disagreements and all unclear decisions were screened by both authors, who made every final inclusion decision. Model temperature could not be set. Inputs, prompts, raw outputs and the run record are in the supplementary material.

What it taught her: the disagreements taught her the most. They pointed straight at the abstracts that needed a human.

Company: sorting complaints for the quality report

Structure it. Tobias takes over the complaint sorting from his colleague who moved sites. "About 400 complaint texts a quarter into eight defect categories. A useful result: a sorting any of us can rerun, and a record that answers the auditor's question for any single complaint, next quarter or in three years."

Understand it. The complaints contain customer names and hospital details, which he removes with a find-and-replace list before anything else. He learns from quality management that the category of a complaint has to be traceable for years, as long as the device's records are kept. A cloud model that's retired in two years can't be rerun then. And the complaint texts, even with names removed, are material the company prefers to keep in the building. Back to Structure it: "A model we keep ourselves, with a fixed seed, so the same input gives the same output on the same machine. And a person who confirms every category before it goes into the report."

Choose tools. LM Studio with an open model downloaded once and stored with the project, because the model file doesn't change and can be kept as long as the records. He records the file name and its checksum. Temperature 0 and a fixed seed, set in LM Studio and written into the record. The alternative is the Claude API with a pinned model, which sorted the tricky cases better in his trial on 30 old complaints. He keeps it as the option if the local model's errors turn out too costly, with its retirement date noted. If the team later wants one shared local model instead of one per laptop, the route is Heidelberg AI Desk (T11-L03).

Build it. The same folder shape: input file with checksum, prompt version, model file name and checksum, LM Studio version, seed and temperature, one raw output per run. The rule: "Same means the same category for every complaint number." He runs the quarter twice on his laptop. The outputs are identical, word for word. His colleague Sara reruns it on her laptop from the folder alone: 3 of 412 categories differ. Same model file, same seed, same settings, different machine. They add one line to the record: "Identical on the same machine. On a second laptop 409 of 412 agreed. The three differences were decided by the QA lead." Every category is still confirmed by a person before it goes into the management review, and the confirmation is a column in the comparison file.

When the auditor asks about complaint 2026-0417, Tobias opens the folder. It holds the input line, the raw output with the model's reason, and the QA lead's confirmation.

In both cases nobody made the model perfectly repeatable. They made everything around it repeatable, measured the rest, and wrote it down.

What goes wrong

Rerunning in the chat app. Symptom: the same prompt gives different answers, and you can't tell why. Fix: use a route that names the exact model and shows the whole request: an API model ID or a local model file. Keep the chat for exploring.

Editing the prompt in place. Symptom: prompt-final.txt holds today's wording, and nobody knows what produced March's output. Fix: give every prompt a version number and never overwrite an old one.

Same file name, different file. Symptom: both runs used abstracts.txt, but someone added three records in between. Fix: record the input's checksum, and have the person rerunning check it first.

Deciding afterwards what counts as the same. Symptom: differences are explained away one by one after you've seen them. Fix: write the comparison rule in the run record before the second run.

Keeping the run you like. Symptom: you rerun until the result matches March, and the other runs disappear. Fix: number every run and keep every raw output. Report all of them.

Trusting temperature zero. Symptom: "it's deterministic", and then a colleague's laptop gives three different answers. Fix: record the settings, test the rerun on a second machine, and report what you actually got.

A model nobody can run any more. Symptom: the cloud model was retired, and the step can't be repeated for the audit. Fix: note the retirement date in the record. For steps that must be repeatable for years, keep a local model file with the project.

Do it yourself (60 minutes)

Structure it (10 min). Pick one AI step from your own work. Write what it does, who will rerun it, and what they need to see. Decide the level: same words, same decisions, or same method.
Understand it (10 min). Run the step twice as you do today, and count the disagreements. Look at a few. Rewrite your goal if they teach you something.
Choose tools (5 min). Pick a pinned cloud model or a local model file, with one line on why. Note whether temperature and seed can be set, and the model's retirement date if it has one.
Build it (25 min). Set up the folder. Save the input and its checksum, the versioned prompt and the run record with the comparison rule. Run twice from the start, keep both raw outputs, compare, and send every disagreement to a human decision.
Test it (10 min). Have your co-author rerun the step from the folder alone. Compare with your rule, and write the Methods or report sentence.

Exit check

Delivered artifact: a shared folder for one AI-assisted step, with:

the input, untouched, and its checksum;
the prompt, with a version number;
the run record: model ID or model file and checksum, software, settings, dates, and the comparison rule written before the rerun;
the raw output of at least two of your runs, unedited;
a co-author's rerun from the folder alone;
the comparison: agreement, every difference, and who decided each one;
a Methods or report sentence stating the agreement and what couldn't be fixed.

It passes when the co-author's rerun is compared on the rule you set beforehand, and the sentence states the real agreement, not the best run.

Rule to remember

Pin what you can, keep every run, and report what still varies.

Record an AI Step So a Co-Author Can Rerun It