Test a Tool on Your Own Work Before Your Team Depends on It | Heidelberg AI Curriculum
T16-L03
Test a Tool on Your Own Work Before Your Team Depends on It
Before colleagues rely on an AI tool's output, test it on 15 to 20 past cases whose right answers you already know. Write the scoring rule first, count first-pass results, repairs and critical failures, and leave a decision with a routing rule, a fallback and a case pack a colleague can rerun.
T16-L03 · Choose tools · Level 3 Builder · 16-minute read, plus 3 hours of doing
The demo that read every certificate
A lab manager watches a colleague drop five supplier certificates into an AI assistant. Out come lot numbers, purity and expiry dates, in a neat table, in seconds. The group decides to use it for every incoming reagent. Six weeks later an auditor asks about an antibody. The inventory says it expires in 2028. The certificate says retest date 2026, and expiry isn't on it at all. The assistant had put one in the place of the other, on every certificate from that supplier.
In a company it's a purchasing team checking supplier order confirmations against their own orders. An assistant compares each confirmation PDF with the order and lists the differences. In the trial it catches everything. In the third month a supplier raises a price in a footnote, the assistant doesn't mention it, and the invoice arrives at the new price.
Both tools did well in the demo. Neither had been tested on the cases that matter: the hard ones, the rare ones, the ones where a wrong answer looks exactly like a right one. You are at Level 3 Builder: once colleagues depend on the output, they stop checking it. This book shows you how to test a tool on your own past work, with answers you already know, before anyone relies on it.
After this you can
Write a one-page workload card that says what the tool must do, for whom, and which failure is unacceptable.
Build a case pack of 15 to 20 past cases with known answers, keeping ordinary and hard cases apart.
Score every run against a rule written before the first run, and count first-pass results, repairs and critical failures.
Turn the failures into a routing rule a colleague can apply to the input, before the output reaches anyone.
Leave a decision note with a fallback, re-test triggers and a case pack a colleague can rerun without you.
The tool you want to depend on, set up the way your team would use it. The lab example tests a Claude project on the group's Team plan: the group already pays for it, it reads PDFs including scans, and the plan doesn't train on the group's content by default. ChatGPT is the usual alternative. For confidential material, use the assistant your organisation has approved.
15 to 20 past cases where someone already worked out the right answer: entries that were checked, decisions that were reviewed.
A spreadsheet for the results, and a shared folder for the pack.
One colleague to rerun two cases, with 30 minutes to spare.
About 3 hours 15 minutes: 16 to read, about 3 hours to do, over a week.
Last verified 23 September 2026. Claude Team costs US$25 per person a month, or US$20 billed annually; by default, Team content isn't used for training. Claude reads scans and charts in PDFs up to 100 pages; longer ones, text only. Details for experts at the end of this book.
The idea in one page
A demo shows what a tool can do. An evaluation shows what it does on your work, including the cases you'd rather not think about. It has five parts, and all five fit in one folder.
Part
What it holds
1. Workload card
The job, the volume, who acts on the output, the one failure you can't accept, and the setup you're testing.
2. Case pack
15 to 20 past cases with their known answers. Ordinary and hard cases in separate groups.
3. Scoring rule
Pass, repair, fail, critical fail, defined before the first run.
4. Results
Every first output, saved, scored and timed.
5. Decision note
Use it, narrow it, or drop it; the routing rule, the fallback, the re-test triggers.
Known answers are the whole trick. Take cases someone has already checked: last quarter's inventory entries, reviewed order confirmations. You don't need to argue about what the right answer is, because it's already in your records. Include:
Ordinary cases, the bulk of real work: ten to twelve.
Hard cases: a scan, a crowded page, an unusual layout, a known trap like retest date: three to five.
Missing cases: the field isn't there. The right output is "not stated".
Out of scope: an input the tool shouldn't process. The right output is to say so.
Write the scoring rule first. Four grades, decided before you look at any output:
Grade
Meaning
Pass
Every required field right. Usable after the normal quick check.
Repair
A visible, small fix is needed. Count the minutes.
Fail
A required field wrong or missing. The person redoes it by hand.
Critical fail
The unacceptable failure from the workload card, and it doesn't look wrong.
Save the first output, every time. Use a fresh chat for each case, the same instructions, the same model. Don't regenerate until it looks right; that's how demos stay perfect. Run the hard cases three times, because the same input can give different outputs.
Count what your team will feel. First-pass rate, the share of runs that pass with no fix at all, on ordinary cases, and separately on hard ones. Repair minutes. Critical fails, which count on their own, however few. Time per accepted item, compared with how long the job takes today. Don't blend it all into one score: one critical fail can outweigh fifty passes.
Turn failures into a routing rule. "Use it, except when it gets the date wrong" helps nobody, because you can't see that from the input. "Certificates with more than one lot go to manual entry" is a rule a colleague can apply before the tool runs. If you can't write a rule someone can apply to the input, keep a full human check on every output, or don't use the tool for this job.
The pack is the Level 3 build. A colleague has to be able to rerun it: same cases, same instructions, same scoring. Write down the tool, plan, model name as shown, instructions version and date. Rerun the pack when any of them changes. Moving the tool into the organisation's systems, and switching tools later, are the next books.
Structure it. Sabine manages a biochemistry lab of twelve. About 30 reagents arrive a month, each with a certificate of analysis. She types lot number, purity, expiry date and storage temperature into the group's inventory spreadsheet, about four minutes per certificate. Her workload card:
Workload card: certificates of analysis -> inventory (v1, 23 Sep 2026)
Job: read one certificate (PDF, sometimes a scan) and return
lot number, purity, expiry date, storage temperature.
Volume: about 30 a month; 60 in the month before an audit.
Who acts: everyone who takes a reagent from the shelf trusts the
expiry date in the inventory. Auditors read it.
Unacceptable: a wrong lot number or expiry date that looks right.
Tested setup: Claude project "CoA reader" on the group's Team plan,
instructions v1, model as shown in the chat, fresh chat
per certificate.
Today: typed by hand, ~4 min each; last audit found 1 typo in 90.
Understand it. She pulls 15 certificates from last year whose inventory entries were checked in the audit, so the right answers are known. She adds four hard ones: a scan of a faxed page, a certificate with two lots on one page, one from the antibody supplier whose certificates give a retest date instead of an expiry date, and one with no storage temperature. And one out-of-scope document: a safety data sheet, which looks similar but isn't a certificate.
Before writing the scoring rule she tries three certificates by hand, just to see the shape of the output. The antibody certificate comes back with the retest date in the expiry column. She checks the inventory: last year she had done the same thing, on that supplier's certificates. The auditor hadn't noticed. The problem was older than the tool. Back to Structure it: the inventory gets a retest date column, the known answers for that supplier are corrected with the group leader, and the workload card now says expiry and retest date are different fields; never copy one into the other.
Choose tools. The candidate is a Claude project, because the group already pays for Team seats and the project keeps the instructions fixed for every chat. The baseline, the way the job is done today and the thing the tool must beat, is her own typing, and it's a real candidate: if the tool can't beat four minutes and one typo in 90, it doesn't get the job. She considers ChatGPT, which some students use, and doesn't add it: a second candidate only makes sense if the first leaves a question open. The results go in a spreadsheet, one row per run. No evaluation software.
Her scoring rule, written before the first run:
Pass: all five fields match the known answer (dates in any format).
Repair: a field is "not stated" when it is stated, or formatted so it
needs retyping. Count minutes.
Fail: a field has a wrong value that someone would notice at once
(e.g. a purity of 9.5%).
Critical: a wrong lot number, expiry or retest date that looks plausible.
Out of scope: pass only if it says this isn't a certificate of analysis.
Build it. The instructions in the project:
CoA reader, instructions v2 (23 Sep 2026)
Read the attached certificate of analysis. Return one table row:
lot | purity | expiry date | retest date | storage temperature
Copy values exactly as printed. If a field isn't on the certificate,
write "not stated". Never put a retest date in the expiry column, or
the other way round. If the page shows more than one lot, write
"MULTIPLE LOTS" and nothing else. If the document isn't a certificate
of analysis, say so and stop.
She runs each of the 20 cases in a fresh chat inside the project, and the five hard and out-of-scope cases three times each: 30 runs. Every first output goes into the results sheet, unchanged, with the time from upload to usable row.
Group
Runs
Pass
Repair
Fail
Critical
Ordinary (15 cases, once each)
15
14
1
0
0
Hard and out of scope (5 cases, 3 times each)
15
9
3
1
2
The critical failures are both on the two-lot certificate: twice out of three runs, the assistant ignores the instruction and mixes the lot number of one with the expiry of the other. The fail is the faxed scan, where it reads a purity of 9.5% for 99.5%. The retest-date certificate passes all three runs, now that the instruction names the trap. The safety data sheet is recognised all three times.
Then the numbers her group will feel. On ordinary certificates, a usable row takes about a minute including her check of lot and expiry against the PDF, against four minutes by hand. The one repair took two minutes. Cost is the Team seat she already has.
The failures turn into a routing rule anyone can apply by looking at the input: certificates with more than one lot, and faxed or photographed pages, are typed by hand. That covers every fail and critical fail in the pack. Her decision note:
Decision: use for single-lot, digital certificates only (23 Sep 2026).
Route to manual entry: more than one lot on the page; scans, faxes,
photos.
Check every time: lot number and expiry/retest date against the PDF,
before saving the row. Other fields: spot check.
Tested: 20 cases, 30 runs, results.xlsx. Ordinary: 14/15 pass,
1 repair. Hard: 2 critical fails, both multi-lot (now routed).
Fallback: type the row by hand from the PDF, as before. The template
is in Inventory/README.
Maintainer: Sabine. Deputy: Tom.
Re-test the whole pack when: the model shown in Claude changes; a new
supplier's format appears; any critical fail in real use; every
6 months.
The rerun. Tom, a postdoc, takes the pack folder: workload card, 20 PDFs with the known answers, instructions, scoring rule, results sheet and decision note. Without asking Sabine, he runs two cases, one ordinary and the two-lot certificate, in fresh chats in the shared project, and scores them with the rule. Ordinary: pass. Two-lot: this time the assistant writes MULTIPLE LOTS, which is a pass under the rule. The sheet already shows this case failing critically in two runs of three, so a pass now doesn't change the result. Either way the routing rule sends two-lot certificates to manual entry, so he types it by hand from the fallback template. One question for Sabine afterwards: is a date written "2027-03" a pass? The scoring rule now says yes.
Where it stops. The group leader asks whether rows could flow straight into the institute's electronic lab notebook. That's an institute system, run by the core facility with its own accounts and approvals. Sabine writes down what the group would need and sends it to them. Her evaluation covers the tool as the group uses it; connecting it to the notebook would need its own test and someone else's approval.
What it taught her: the most serious error in the pack was hers, from last year. The evaluation found it because it compared outputs with known answers, not with what looked plausible.
Company: order confirmations against purchase orders
Structure it. Marco leads a purchasing team of four at a lab-equipment maker. About 250 supplier order confirmations arrive a month as PDFs. Someone compares each with the purchase order: item, quantity, price, delivery date. "Unacceptable: a changed price or delivery date that nobody flags."
Understand it. He pulls 20 past confirmations whose differences are known, because each was resolved with the supplier: 12 that matched, 5 with a changed delivery date, 2 with a changed price, 1 with a price change in a footnote. He also looks at how the job is done today and finds something he'd overlooked: a third of suppliers send confirmations as structured files, which a program can read field by field, and the company's ERP system, its central software for orders and stock, already compares those with the order, with no AI at all. The real workload is the PDFs. His card changes to say so.
Choose tools. Two candidates. The assistant the company has approved for supplier documents, with a saved instruction to list every difference in item, quantity, price and date. And the baseline, a person comparing by hand, about five minutes per confirmation.
Build it. He writes the scoring rule first; a missed price or date change is critical. He runs the 20 cases once and the 8 with differences twice more. The assistant finds every date change and both ordinary price changes, in every run. It misses the footnote price change in two of three runs. It also flags differences that aren't there, like "EUR" against "€", in four ordinary cases; each takes a minute to dismiss.
He can't write a routing rule for footnotes: nobody can tell from the outside which confirmation has one. So the decision is narrower than he'd hoped. The assistant marks date changes, which were the ones people missed most. Prices are still compared by hand, line by line. That saves about two minutes per confirmation instead of four, and it's two minutes he can trust. His deputy reruns the footnote case and a matching one from the pack and gets the same scores. Re-test triggers go in the note, as in Sabine's.
Where it stops. Finance asks whether the assistant could read confirmations straight from the purchasing mailbox and write the differences into the ERP. That's a connection between company systems. Marco hands the request, with his evaluation pack, to IT.
In both cases the evaluation ended with a narrower decision than the demo suggested, and that narrower decision was the one colleagues could safely depend on.
What goes wrong
Testing on easy cases.Symptom: 20 of 20 pass, and the first real scan fails. Fix: take cases from your records, including the ugly ones, and keep ordinary and hard cases in separate rows so neither hides the other.
The known answer is wrong.Symptom: the tool "fails" on a case where it's actually right. Fix: when a failure surprises you, check the known answer first. Correct it with whoever decides, and write down why.
Regenerating until it's right.Symptom: every result in the sheet is perfect, and real use isn't. Fix: save the first output of every run. If retrying is part of the real workflow, count every try.
One average.Symptom: "93% accurate", and the 7% includes the failure you can't accept. Fix: count critical fails on their own. One is enough to route a case type to manual work.
A rule nobody can apply.Symptom: "don't trust it on tricky ones". Fix: write a routing rule based on something visible in the input: the number of lots, a scan, a supplier. If you can't, check every output or drop the tool for this job.
Testing a different setup from the one you'll use.Symptom: the evaluation used your personal account; the team uses a shared project with other instructions. Fix: write the exact setup on the workload card and test that.
No re-test.Symptom: the model behind the tool changes, and the pack is never run again. Fix: re-test triggers in the decision note, and a date in the maintainer's calendar.
Do it yourself (3 hours, over a week)
Structure it (20 min). Write the workload card: job, volume, who acts on the output, the unacceptable failure, the exact setup, and how the job is done today.
Understand it (40 min). Collect 15 to 20 past cases with checked answers, including three to five hard ones, one missing and one out of scope. Try three by hand to see the output. If that shows something about the job itself, fix the card and the known answers first.
Choose tools (15 min). Name the candidate and the baseline. Write the scoring rule. Set up the results sheet.
Build it: run and score (60 min). One fresh session per case. Hard cases three times. Save every first output. Score, time and count.
Build it: decide and hand over (45 min). Write the routing rule and the decision note with the fallback, maintainer, deputy and re-test triggers. Ask a colleague to rerun two cases, one of them hard, and the fallback from the pack alone. Fix what confused them.
Exit check
What you hand in: one evaluation pack for a tool and a recurring job, with:
the workload card, including the unacceptable failure and the exact setup tested;
a case pack of at least 15 cases with known answers, ordinary and hard cases marked;
the scoring rule, written before the first run;
the results, with every first output saved, and counts for ordinary cases, hard cases and critical fails;
a one-page decision note with the routing rule, fallback, maintainer, deputy and re-test triggers.
It passes when a colleague reruns two cases, one of them hard, and the fallback from the pack alone, and reaches the same scores.
Rule to remember
Test the tool on answers you already know, and let the worst case, not the average, set the rule.
Details for experts
Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.
What
Details
What it means for you
Claude Team price
US$25 per Standard seat (one person's account) a month billed monthly, or US$20 billed annually, before tax. For teams of 2 to 150.
The seat the lab already pays for covers the whole evaluation.
Training on your content
Anthropic states that it doesn't train models on Team content by default.
"By default" means a setting decides it. Ask whoever runs your team account before you upload confidential certificates.
File types
Documents: PDF, DOCX (Word), CSV, TXT, HTML, ODT, RTF, EPUB, JSON and XLSX (Excel). Images: JPEG, PNG, GIF and WebP.
Certificates arrive as PDFs or phone photos; both upload.
File limits
Up to 500 MB per file and 20 files per chat.
One certificate per fresh chat is far inside the limit.
Scans and charts in PDFs
Claude reads the images and charts in PDFs of up to 100 pages. Beyond that it reads the text only. It doesn't accept PDFs of more than 1,000 pages.
A scanned page in a very long PDF gives nothing to read. Upload scans as separate, short files.
Projects
On every plan. Sharing with colleagues needs Team or Enterprise.
The deputy's rerun needs the shared project, so both of you must be on the team plan.
If these details change
The method doesn't: the case pack tests the tool you have, as it's set up today.
Rerun the pack when the plan, model or instructions change, as the decision note says.
Claude: Taught. The project the lab example tests.
ChatGPT: Catalogued. The usual alternative candidate.
Microsoft Copilot: Catalogued. Often the assistant already in a company's office software; test it the same way.
Promptfoo: Catalogued. Runs a case pack automatically when the tool can be called from a script.
Watch
17 minTest Driven PROMPT Engineering: Using Promptfoo to COMPARE Prompts, LLMs, and Providers.
Test Driven PROMPT Engineering: Using Promptfoo to COMPARE Prompts, LLMs, and Providers.IndyDevDan · 17 min
T03-L03: See file-based Promptfoo cases, assertions and uncached comparisons. The 2023 demonstration compares two OpenAI model configurations; follow the book's pinned setup, critical-case rules and human review. Its small cost and speed examples are historical, not production guarantees.
T05-L02: See how explicit test cases and assertions make model outputs checkable. This developer-oriented Promptfoo example evaluates outputs offline; keep the book's runtime checks on every item. Its cost and speed comparisons are historical examples.
T16-L03: See repeatable cases and explicit assertions turn model comparisons into checks you can rerun. This optional technical illustration does not require replacing the book's spreadsheet with Promptfoo; cost and speed ratios are historical.
I Tested Sonnet 5.5 vs Opus 5.5. What You Need to Know.
I Tested Sonnet 5.5 vs Opus 5.5. What You Need to Know.Video · Nate Herk · Sep 29, 2026
In a video by Nate Herk, Claude Sonnet 5.5 is tested against Opus 5.5 across seven real workflows (landing pages, messy data turned into spreadsheets, motion design/video editing, live dashboards, project planning) to determine when to use each model. Sonnet 5.5 is described as a clear upgrade over Sonnet 5, running 30% faster and costing 30% less for most work. Pricing: Sonnet 5.5 costs $2 per million input tokens and $10 per million output tokens, while Opus 5.5 costs $4 and $20 respectively—roughly double. Drawing on Claude's own guidance, Sonnet suits well-scoped, high-volume, clearly-specified tasks with a way to verify results (everyday coding, bug fixes, polished documents, slides, spreadsheets), whereas Opus suits complex work requiring judgment, creativity, and the most intelligence. In a landing-page demo for 'Perk Form' using identical prompts with both on high effort, Opus produced the better overall output by using real generated product images within brand guidelines, though Sonnet's hero section was stronger. Sonnet ran 28 minutes with 20M input/134K output tokens; Opus ran 52 minutes with 37M input/148K output tokens, costing about double.
