Reproducibility with AI in the loop

This mini-book is for Level 2 Power user. An AI-assisted result may now enter shared research or company work. A colleague must be able to identify what you ran, repeat the attempt, and see whether the answer changed. A successful rerun does not remove your responsibility to validate the result.

2. September does not reproduce March

In March, you used an approved assistant to classify forty short records. You saved the final labels and wrote "AI classification" in your notes. In September, a reviewer asks you to rerun the step. You paste what looks like the same prompt into the same product and receive four different labels.

The interface still shows the same friendly model name. You did not record whether that name referred to a dated release, which settings were active, the exact input file, or the prompt before its final edits. There is no defensible way to tell whether model drift, input drift, prompt drift, hidden retrieval, or ordinary generation variability caused the change.

You cannot repair March after the fact. You can make the next run inspectable: freeze the inputs and prompt, record the model and settings the service exposes, save the raw output, define what "same answer" means, rerun, and report the delta without hiding it.

3. After this you can

• Record a model or route, available version identifier, prompt version, parameters, input hash, date, and raw output for one AI-assisted step.
• Pin the controllable parts of a run and mark hidden or unavailable values explicitly.
• Rerun a classification step against frozen synthetic or public input.
• Compare raw and normalized outputs against a predeclared acceptance test.
• Report the observed difference and the limit of any reproducibility claim.

4. Prerequisites

• T15-L01 - AI in your research workflow, including its use-log and disclosure boundary.
• T10-L02 - Clean a messy export, including the principle that raw input stays unchanged and transformations are repeatable.
• An approved assistant or model interface that lets you retain outputs. An API is optional.
• A text editor, a file-hashing tool, and a safe folder for prompts, inputs, outputs, and the run record.
• Four or more public, synthetic, or explicitly approved records with a classification rule you can check yourself.

Do not upload unpublished results, participant or patient data, peer-review material, personal data, credentials, confidential company records, or licensed text to an unapproved service. A hash is an identifier, not anonymisation: keep the underlying data within its approved boundary.

5. The idea in one page

Reproducibility starts with a declared target. Byte reproducibility means the saved output bytes are identical. Decision reproducibility means a defined result, such as the ordered label vector, is identical after an agreed normalization. Method reproducibility means another person can rerun the recorded procedure, even if a non-deterministic service returns a different answer. Name the level you tested; never let "reproducible" silently mean all three.

For each AI-assisted step, retain this minimum record:

A SHA-256 hash detects changed bytes; it does not prove that the file is correct, safe, or honestly described. Keep the file and its provenance. Hash the exact file used, not a later copy. For example, PowerShell can run Get-FileHash -Algorithm SHA256 .\input.json; use an approved equivalent on another system.

Pin what the service permits: an immutable input, versioned prompt, dated model release, explicit parameters, disabled unrecorded tools, and a known software environment. Then accept the limit. Temperature zero chooses a less varied sampling policy; it is not a universal identity guarantee. A seed may support repeatability only within the provider's implementation. Ties, hardware, routing, safety layers, retrieval indexes, and service updates can still alter output. If a consumer interface exposes only a moving display name, record the name and exact version not exposed; do not claim a model-pinned rerun.

Define the comparison before seeing the second output. Normalization may parse JSON and compare ordered id and label values while ignoring whitespace. It must not silently correct labels, drop records, or turn an invalid answer into a pass.

6. The worked example: one protocol, two settings

The Lab and Company lanes below are synthetic and deliberately parallel. Both use four short records, the same labels, the same fixed prompt, and the same acceptance test. The model and outputs are course fixtures, not evidence about a real provider.

Freeze the prompt

Save this exact UTF-8 text without a trailing newline as prompt P-01:

Classify each supplied record as REVIEW or NO_REVIEW.
Use REVIEW only if the text explicitly says the required field is missing or the value is outside the stated range.
Otherwise use NO_REVIEW. Do not infer from an empty note.
Return JSON only: [{"id":"...","label":"REVIEW|NO_REVIEW"}]
Preserve input order. Do not add keys or explanations.

Its SHA-256 hash is 2ad90c96b585356599da9b1d634d299eb2922285a7b6a7a4e9c2939aa4ab36b8. If your saved file produces another hash, investigate encoding, line endings, trailing whitespace, or changed text before running.

The acceptance test is declared now: parse valid JSON; require all four IDs once in input order; allow only the two named labels; compare the ordered label vector exactly. Also compare the raw output hashes separately. Human review checks each label against the literal rule.

Lab framing: synthetic instrument notes

Save this one-line UTF-8 JSON without a trailing newline as lab-input.json:

[{"id":"L01","text":"Sensor check complete; reference reading is present."},{"id":"L02","text":"Reference reading is missing; hold this batch for review."},{"id":"L03","text":"Operator note is blank; reference reading is present."},{"id":"L04","text":"Reference reading is outside the stated 9-11 range."}]

Its SHA-256 hash is b6069548515cacf7f71c67dc958c942d891a9c44fcf6d30f8060ece5cb9f7542. The independently checked target vector is NO_REVIEW, REVIEW, NO_REVIEW, REVIEW. In particular, the blank operator note in L03 is not the required reference reading.

Company framing: synthetic invoice notes

Save this parallel one-line input as company-input.json:

[{"id":"C01","text":"Invoice check complete; purchase-order number is present."},{"id":"C02","text":"Purchase-order number is missing; hold this invoice for review."},{"id":"C03","text":"Reviewer note is blank; purchase-order number is present."},{"id":"C04","text":"Invoice total is outside the stated 90-110 range."}]

Its SHA-256 hash is 898574f8f3e662d8beea8113348c382254c570ff25d80bba6ff8b57bf2b74e23. The checked target vector is again NO_REVIEW, REVIEW, NO_REVIEW, REVIEW. The blank reviewer note does not make the purchase-order number missing.

Complete one run record

The course fixture records a March run and a September rerun in one reproducibility record:

For the Lab lane, the fixture outputs saved without trailing newlines are:

Run A (2faca62ae67dd55da06a030501a15b8771647b68bd28bad4b2a47efd65334d25)

[{"id":"L01","label":"NO_REVIEW"},{"id":"L02","label":"REVIEW"},{"id":"L03","label":"NO_REVIEW"},{"id":"L04","label":"REVIEW"}]

Run B (941c16790deba4762997761f98c84d3b6c4894fecbf361d50bff66c9d3933ad7)

[
  {"id":"L01","label":"NO_REVIEW"},
  {"id":"L02","label":"REVIEW"},
  {"id":"L03","label":"NO_REVIEW"},
  {"id":"L04","label":"REVIEW"}
]

The Company lane is exactly parallel:

Run A (5a8a43e6741c842374301932cddd918d57e6117b0fdca78d0f281c2e888f97d3)

[{"id":"C01","label":"NO_REVIEW"},{"id":"C02","label":"REVIEW"},{"id":"C03","label":"NO_REVIEW"},{"id":"C04","label":"REVIEW"}]

Run B (10befc52aa2d00426d1ac94387aae5b4ac5b5fc9e00988db04522840efb86668)

[
  {"id":"C01","label":"NO_REVIEW"},
  {"id":"C02","label":"REVIEW"},
  {"id":"C03","label":"NO_REVIEW"},
  {"id":"C04","label":"REVIEW"}
]

The raw hashes differ, so byte reproducibility is FAIL in both lanes. The Lab outputs parse to L01:NO_REVIEW, L02:REVIEW, L03:NO_REVIEW, L04:REVIEW, and the Company outputs produce the corresponding four labels. Decision reproducibility is therefore PASS (4/4) in each lane.

The reported delta is not "no difference." It is: model release changed; raw serialization changed; 0/4 normalized labels changed. Because the route resolved to different releases, this is not a model-pinned replication even though the decision answer reproduced. A Methods sentence can therefore say:

We classified four synthetic records using prompt P-01 and the recorded parameters. The September rerun used identical input and prompt hashes but the service reported a different model release. Raw output bytes differed in formatting; the predeclared ordered label vector was unchanged (4/4 agreement). Inputs, prompt, raw outputs, normalization rule, and run metadata were retained.

Replace every fixture value with observed values from your run. If your interface does not reveal a returned release, write exact version not exposed and narrow the final sentence accordingly. Do not copy the fixture dates or model names into a real Methods section.

7. What goes wrong

The prompt is saved but the model version is not

Symptom: the record says same assistant although the display name may route to another release.

Fix: record provider or runtime, requested identifier, returned release if exposed, and run date. If no exact release is visible, state that limitation and avoid a model-pinned claim.

The prompt is edited in place

Symptom: final-prompt.txt contains today's wording, but no one can recover the wording used for the first output.

Fix: assign a prompt version, retain each used version, and hash it. Use the version history practice from T03-L04 - Build a prompt library.

Temperature zero is treated as a guarantee

Symptom: any changed output is dismissed as impossible or blamed on the operator.

Fix: record temperature and every exposed control, then test actual reruns. Report observed identity or difference; do not infer determinism from one setting.

The input has a familiar filename but no hash

Symptom: both runs say records-final.json, yet a row, newline, or correction changed between dates.

Fix: preserve the immutable input used by each run and compare its cryptographic hash before comparing outputs.

Normalization hides a real disagreement

Symptom: a script sorts, drops invalid rows, or maps an unexpected label to the nearest allowed value and then reports a pass.

Fix: predeclare normalization, fail on missing or duplicate IDs and invalid labels, and retain raw outputs for inspection.

Only the preferred rerun survives

Symptom: repeated attempts continue until one matches March, and the mismatching attempts disappear.

Fix: number every attempt, retain all outputs, and report the attempt count plus every observed delta. A lucky match is not an honest replication.

8. Do it yourself: rerun and report in 45 minutes

Minutes 0-7: choose one bounded Lab or Company classification using four to ten public, synthetic, or explicitly approved records. Write the literal label rule and the expected IDs. Confirm that a human can check the result.

Minutes 7-14: save an immutable input and calculate its SHA-256 hash. Save the exact prompt under a version ID and hash it too. Do not normalize either file before hashing the bytes that will actually be sent.

Minutes 14-20: declare the acceptance test. State whether you require byte identity, ordered label identity, or another exact decision-level result. Write which differences normalization may ignore and which must fail.

Minutes 20-27: record the service or runtime, requested model, returned model/version if exposed, UTC time, parameters, tools, retrieval state, and wrapper version. Run once and retain the unedited raw output.

Minutes 27-34: start a clean rerun with the same input bytes, prompt bytes, exposed model identifier, parameters, and environment. Retain the second raw output even if it is invalid or surprising.

Minutes 34-41: compare input and prompt hashes first, raw output hashes second, and the normalized answer third. Manually check each classification. Count changed, missing, duplicate, and invalid labels.

Minutes 41-45: write one result sentence naming what was fixed, what changed, the measured delta, and what was not exposed or controlled. Mark the test PASS only against the acceptance criterion you declared before rerunning.

9. Exit check

Deliver exactly one artifact: one reproducibility record for a single AI-assisted classification step, containing its purpose and owner; immutable input name and hash; full prompt, version and hash; service or runtime; requested and returned model/version fields; UTC dates; exposed parameters, tools and environment; both unedited outputs; the predeclared normalization and acceptance test; human validation; and the measured delta.

It passes when another person can reconstruct the attempted rerun from the record, verify that input and prompt bytes match, inspect both raw outputs, repeat the comparison, and reach the same pass or fail decision. A record can pass the documentation test while reporting a failed rerun, provided the failure and unavailable controls are explicit. It fails if a changed label, hidden attempt, unknown input, or unavailable model version is silently presented as identical.

10. Rule to remember

Record the version, or you recorded nothing.

11. Further reading & tools

• Taught: T15-L01 - AI in your research workflow - establishes the use log, source checks, disclosure, and human responsibility that this record extends.
• Taught: T10-L02 - Clean a messy export - preserves raw input and makes transformations repeatable before AI enters the step.
• Taught: T03-L04 - Build a prompt library - versions prompt text instead of editing the only copy in place.
• Catalogued: NIST AI RMF: Generative AI Profile (opens in a new tab) - risk-management guidance for documenting, measuring, and monitoring generative AI systems.
• Catalogued: Python hashlib documentation (opens in a new tab) - one standard-library route for calculating SHA-256 when an approved Python environment is available.
• Catalogued: Git documentation (opens in a new tab) - a version-control option for retaining prompt and run-record history; a commit does not replace data approval or raw-output retention.
• Catalogued: Tools index - compare approved interfaces and runtimes; verify current version visibility, parameter support, retention, and data handling before use.