Test a Tool on Your Own Work Before Your Team Depends on It

T16-L03 · Choose tools · Level 3 Builder · 16-minute read, plus 3 hours of doing

The demo that read every certificate

A lab manager watches a colleague drop five supplier certificates into an AI assistant. Out come lot numbers, purity and expiry dates, in a neat table, in seconds. The group decides to use it for every incoming reagent. Six weeks later an auditor asks about an antibody. The inventory says it expires in 2028. The certificate says retest date 2026, and expiry isn't on it at all. The assistant had put one in the place of the other, on every certificate from that supplier.

In a company it's a purchasing team checking supplier order confirmations against their own orders. An assistant compares each confirmation PDF with the order and lists the differences. In the trial it catches everything. In the third month a supplier raises a price in a footnote, the assistant doesn't mention it, and the invoice arrives at the new price.

Both tools did well in the demo. Neither had been tested on the cases that matter: the hard ones, the rare ones, the ones where a wrong answer looks exactly like a right one. You are at Level 3 Builder: once colleagues depend on the output, they stop checking it. This book shows you how to test a tool on your own past work, with answers you already know, before anyone relies on it.

After this you can

• Write a one-page workload card that says what the tool must do, for whom, and which failure is unacceptable.
• Build a case pack of 15 to 20 past cases with known answers, keeping ordinary and hard cases apart.
• Score every run against a rule written before the first run, and count first-pass results, repairs and critical failures.
• Turn the failures into a routing rule a colleague can apply to the input, before the output reaches anyone.
• Leave a decision note with a fallback, re-test triggers and a case pack a colleague can rerun without you.

Before you start

• Books: T16-L02 for choosing a tool your team can use and afford. T01-L02 for checking an output against its source. T03-L03 if the thing you're testing is a prompt you can run from files.
• The tool you want to depend on, set up the way your team would use it. The lab example tests a Claude project on the group's Team plan: the group already pays for it, it reads PDFs including scans, and the plan doesn't train on the group's content by default. ChatGPT is the usual alternative. For confidential material, use the assistant your organisation has approved.
• 15 to 20 past cases where someone already worked out the right answer: entries that were checked, decisions that were reviewed.
• A spreadsheet for the results, and a shared folder for the pack.
• One colleague to rerun two cases, with 30 minutes to spare.
• About 3 hours 15 minutes: 16 to read, about 3 hours to do, over a week.

Last verified 23 September 2026. Claude Team costs US$25 per person a month, or US$20 billed annually; by default, Team content isn't used for training. Claude reads scans and charts in PDFs up to 100 pages; longer ones, text only. Details for experts at the end of this book.

The idea in one page

A demo shows what a tool can do. An evaluation shows what it does on your work, including the cases you'd rather not think about. It has five parts, and all five fit in one folder.

Known answers are the whole trick. Take cases someone has already checked: last quarter's inventory entries, reviewed order confirmations. You don't need to argue about what the right answer is, because it's already in your records. Include:

• Ordinary cases, the bulk of real work: ten to twelve.
• Hard cases: a scan, a crowded page, an unusual layout, a known trap like retest date: three to five.
• Missing cases: the field isn't there. The right output is "not stated".
• Out of scope: an input the tool shouldn't process. The right output is to say so.

Write the scoring rule first. Four grades, decided before you look at any output:

Save the first output, every time. Use a fresh chat for each case, the same instructions, the same model. Don't regenerate until it looks right; that's how demos stay perfect. Run the hard cases three times, because the same input can give different outputs.

Count what your team will feel. First-pass rate on ordinary cases, and separately on hard ones. Repair minutes. Critical fails, which count on their own, however few. Time per accepted item, compared with how long the job takes today. Don't blend it all into one score: one critical fail can outweigh fifty passes.

Turn failures into a routing rule. "Use it, except when it gets the date wrong" helps nobody, because you can't see that from the input. "Certificates with more than one lot go to manual entry" is a rule a colleague can apply before the tool runs. If you can't write a rule someone can apply to the input, keep a full human check on every output, or don't use the tool for this job.

The pack is the Level 3 build. A colleague has to be able to rerun it: same cases, same instructions, same scoring. Write down the tool, plan, model name as shown, instructions version and date. Rerun the pack when any of them changes. Moving the tool into the organisation's systems, and switching tools later, are the next books.

Worked example: two decisions, four quadrants

Both follow the four quadrants from T01-L01. Follow the one closer to your work. The people, cases and counts are examples written for this book.

Lab: reagent certificates into the inventory

Structure it. Sabine manages a biochemistry lab of twelve. About 30 reagents arrive a month, each with a certificate of analysis. She types lot number, purity, expiry date and storage temperature into the group's inventory spreadsheet, about four minutes per certificate. Her workload card:

Workload card: certificates of analysis -> inventory (v1, 23 Sep 2026)
Job:         read one certificate (PDF, sometimes a scan) and return
             lot number, purity, expiry date, storage temperature.
Volume:      about 30 a month; 60 in the month before an audit.
Who acts:    everyone who takes a reagent from the shelf trusts the
             expiry date in the inventory. Auditors read it.
Unacceptable: a wrong lot number or expiry date that looks right.
Tested setup: Claude project "CoA reader" on the group's Team plan,
             instructions v1, model as shown in the chat, fresh chat
             per certificate.
Today:       typed by hand, ~4 min each; last audit found 1 typo in 90.

Understand it. She pulls 15 certificates from last year whose inventory entries were checked in the audit, so the right answers are known. She adds four hard ones: a scan of a faxed page, a certificate with two lots on one page, one from the antibody supplier whose certificates give a retest date instead of an expiry date, and one with no storage temperature. And one out-of-scope document: a safety data sheet, which looks similar but isn't a certificate.

Before writing the scoring rule she tries three certificates by hand, just to see the shape of the output. The antibody certificate comes back with the retest date in the expiry column. She checks the inventory: last year she had done the same thing, on that supplier's certificates. The auditor hadn't noticed. The problem was older than the tool. Back to Structure it: the inventory gets a retest date column, the known answers for that supplier are corrected with the group leader, and the workload card now says expiry and retest date are different fields; never copy one into the other.

That's the loop from T01-L01: trying the tool taught her something about the job, and the job changed before the evaluation began.

Choose tools. The candidate is a Claude project, because the group already pays for Team seats and the project keeps the instructions fixed for every chat. The baseline is her own typing, and it's a real candidate: if the tool can't beat four minutes and one typo in 90, it doesn't get the job. She considers ChatGPT, which some students use, and doesn't add it: a second candidate only makes sense if the first leaves a question open. The results go in a spreadsheet, one row per run. No evaluation software.

Her scoring rule, written before the first run:

Pass:     all five fields match the known answer (dates in any format).
Repair:   a field is "not stated" when it is stated, or formatted so it
          needs retyping. Count minutes.
Fail:     a field has a wrong value that someone would notice at once
          (e.g. a purity of 9.5%).
Critical: a wrong lot number, expiry or retest date that looks plausible.
Out of scope: pass only if it says this isn't a certificate of analysis.

Build it. The instructions in the project:

CoA reader, instructions v2 (23 Sep 2026)
Read the attached certificate of analysis. Return one table row:
lot | purity | expiry date | retest date | storage temperature
Copy values exactly as printed. If a field isn't on the certificate,
write "not stated". Never put a retest date in the expiry column, or
the other way round. If the page shows more than one lot, write
"MULTIPLE LOTS" and nothing else. If the document isn't a certificate
of analysis, say so and stop.

She runs each of the 20 cases in a fresh chat inside the project, and the five hard and out-of-scope cases three times each: 30 runs. Every first output goes into the results sheet, unchanged, with the time from upload to usable row.

The critical failures are both on the two-lot certificate: twice out of three runs, the assistant ignores the instruction and mixes the lot number of one with the expiry of the other. The fail is the faxed scan, where it reads a purity of 9.5% for 99.5%. The retest-date certificate passes all three runs, now that the instruction names the trap. The safety data sheet is recognised all three times.

Then the numbers her group will feel. On ordinary certificates, a usable row takes about a minute including her check of lot and expiry against the PDF, against four minutes by hand. The one repair took two minutes. Cost is the Team seat she already has.

The failures turn into a routing rule anyone can apply by looking at the input: certificates with more than one lot, and faxed or photographed pages, are typed by hand. That covers every fail and critical fail in the pack. Her decision note:

Decision: use for single-lot, digital certificates only (23 Sep 2026).
Route to manual entry: more than one lot on the page; scans, faxes,
  photos.
Check every time: lot number and expiry/retest date against the PDF,
  before saving the row. Other fields: spot check.
Tested: 20 cases, 30 runs, results.xlsx. Ordinary: 14/15 pass,
  1 repair. Hard: 2 critical fails, both multi-lot (now routed).
Fallback: type the row by hand from the PDF, as before. The template
  is in Inventory/README.
Maintainer: Sabine. Deputy: Tom.
Re-test the whole pack when: the model shown in Claude changes; a new
  supplier's format appears; any critical fail in real use; every
  6 months.

The rerun. Tom, a postdoc, takes the pack folder: workload card, 20 PDFs with the known answers, instructions, scoring rule, results sheet and decision note. Without asking Sabine, he runs two cases, one ordinary and the two-lot certificate, in fresh chats in the shared project, and scores them with the rule. Ordinary: pass. Two-lot: the assistant writes MULTIPLE LOTS this time, which is a pass under the rule, but the routing rule sends it to manual entry anyway. He types it by hand from the fallback template. His scores match the sheet's; the difference on the two-lot case is one the sheet already shows. One question for Sabine afterwards: is a date written "2027-03" a pass? The scoring rule now says yes.

Where it stops. The group leader asks whether rows could flow straight into the institute's electronic lab notebook. That's an institute system, run by the core facility with its own accounts and approvals. Sabine writes down what the group would need and sends it to them. Her evaluation covers the tool as the group uses it; connecting it to the notebook would need its own test and someone else's approval.

What it taught her: the most serious error in the pack was hers, from last year. The evaluation found it because it compared outputs with known answers, not with what looked plausible.

Company: order confirmations against purchase orders

Structure it. Marco leads a purchasing team of four at a lab-equipment maker. About 250 supplier order confirmations arrive a month as PDFs. Someone compares each with the purchase order: item, quantity, price, delivery date. "Unacceptable: a changed price or delivery date that nobody flags."

Understand it. He pulls 20 past confirmations whose differences are known, because each was resolved with the supplier: 12 that matched, 5 with a changed delivery date, 2 with a changed price, 1 with a price change in a footnote. He also looks at how the job is done today and finds something he'd overlooked: a third of suppliers send confirmations as structured files, and the company's ERP system, its central software for orders and stock, already compares those with the order, with no AI at all. The real workload is the PDFs. His card changes to say so.

Choose tools. Two candidates. The assistant the company has approved for supplier documents, with a saved instruction to list every difference in item, quantity, price and date. And the baseline, a person comparing by hand, about five minutes per confirmation.

Build it. He writes the scoring rule first; a missed price or date change is critical. He runs the 20 cases once and the 8 with differences twice more. The assistant finds every date change and both ordinary price changes, in every run. It misses the footnote price change in two of three runs. It also flags differences that aren't there, like "EUR" against "€", in four ordinary cases; each takes a minute to dismiss.

He can't write a routing rule for footnotes: nobody can tell from the outside which confirmation has one. So the decision is narrower than he'd hoped. The assistant marks date changes, which were the ones people missed most. Prices are still compared by hand, line by line. That saves about two minutes per confirmation instead of four, and it's two minutes he can trust. His deputy reruns the footnote case and a matching one from the pack and gets the same scores. Re-test triggers go in the note, as in Sabine's.

Where it stops. Finance asks whether the assistant could read confirmations straight from the purchasing mailbox and write the differences into the ERP. That's a connection between company systems. Marco hands the request, with his evaluation pack, to IT.

In both cases the evaluation ended with a narrower decision than the demo suggested, and that narrower decision was the one colleagues could safely depend on.

What goes wrong

Testing on easy cases. Symptom: 20 of 20 pass, and the first real scan fails. Fix: take cases from your records, including the ugly ones, and keep ordinary and hard cases in separate rows so neither hides the other.

The known answer is wrong. Symptom: the tool "fails" on a case where it's actually right. Fix: when a failure surprises you, check the known answer first. Correct it with whoever decides, and write down why.

Regenerating until it's right. Symptom: every result in the sheet is perfect, and real use isn't. Fix: save the first output of every run. If retrying is part of the real workflow, count every try.

One average. Symptom: "93% accurate", and the 7% includes the failure you can't accept. Fix: count critical fails on their own. One is enough to route a case type to manual work.

A rule nobody can apply. Symptom: "don't trust it on tricky ones". Fix: write a routing rule based on something visible in the input: the number of lots, a scan, a supplier. If you can't, check every output or drop the tool for this job.

Testing a different setup from the one you'll use. Symptom: the evaluation used your personal account; the team uses a shared project with other instructions. Fix: write the exact setup on the workload card and test that.

No re-test. Symptom: the model behind the tool changes, and the pack is never run again. Fix: re-test triggers in the decision note, and a date in the maintainer's calendar.

Do it yourself (3 hours, over a week)

1. Structure it (20 min). Write the workload card: job, volume, who acts on the output, the unacceptable failure, the exact setup, and how the job is done today.
2. Understand it (40 min). Collect 15 to 20 past cases with checked answers, including three to five hard ones, one missing and one out of scope. Try three by hand to see the output. If that shows something about the job itself, fix the card and the known answers first.
3. Choose tools (15 min). Name the candidate and the baseline. Write the scoring rule. Set up the results sheet.
4. Build it: run and score (60 min). One fresh session per case. Hard cases three times. Save every first output. Score, time and count.
5. Build it: decide and hand over (45 min). Write the routing rule and the decision note with the fallback, maintainer, deputy and re-test triggers. Ask a colleague to rerun two cases, one of them hard, and the fallback from the pack alone. Fix what confused them.

Exit check

Delivered artifact: one evaluation pack for a tool and a recurring job, with:

• the workload card, including the unacceptable failure and the exact setup tested;
• a case pack of at least 15 cases with known answers, ordinary and hard cases marked;
• the scoring rule, written before the first run;
• the results, with every first output saved, and counts for ordinary cases, hard cases and critical fails;
• a one-page decision note with the routing rule, fallback, maintainer, deputy and re-test triggers.

It passes when a colleague reruns two cases, one of them hard, and the fallback from the pack alone, and reaches the same scores.

Rule to remember

Test the tool on answers you already know, and let the worst case, not the average, set the rule.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T16-L02 · Choose tools your team can use, and afford: picking the candidate before you test it.
• T16-L04 · Replace a tool without losing your work: when a re-test says the tool no longer passes.
• T16-L05 · Keep your toolset useful: looking after the tools once they're in use.
• T03-L03 · Put a test set under the prompt your team depends on: the same method, run automatically, when the job is a prompt.
• T01-L02 · Check AI output before your team relies on it: the check behind every known answer.
• T17-L03 · Explore what already exists before you build: finding the candidates, and the baseline that might already do the job.

Tools

• Claude: Taught. The project the lab example tests.
• ChatGPT: Catalogued. The usual alternative candidate.
• Microsoft Copilot: Catalogued. Often the assistant already in a company's office software; test it the same way.
• Promptfoo: Catalogued. Runs a case pack automatically when the tool can be called from a script.