Run a Pilot with Real Users and Real Data | Heidelberg AI Curriculum
T17-L04
Run a Pilot with Real Users and Real Data
Test a solution on real work before your lab or company depends on it. Set the pilot's scope, success criteria and data permissions before day one, keep a way back, measure what matters, and end with a written go, change or stop decision that follows rules you set in advance.
T17-L04 · Structure & understand · Level 4 Integrator · 18-minute read, plus 4 hours of doing and a 4-week pilot
It worked on the test data
Your group tested an assistant that turns bench notes into entries for the institute's sample inventory. On last year's notes it got nearly everything right. So on Monday three colleagues start using it for real. By Thursday one sample has been entered in box 14 instead of box 41, and a colleague spends forty minutes at the −80 °C freezer with the door open. Someone has pasted a list of clinical samples, with their pseudonymised patient codes (codes that stand in for the patients' names), into a personal chatbot account because the approved one was slow that morning. And when the PI, the group leader, asks at the end of the month whether it's working, nobody can say, because nobody wrote down what "working" meant.
In a company it's a sales team trying AI-drafted replies to quote requests inside the CRM, the customer relationship system where sales keeps its customers and their requests. Replies go out faster. Then a customer accepts a quote with last year's price on it, and the company has to honour it or lose the customer.
You are at Level 4 Integrator. Real systems and real data are now in the loop: the institute's inventory, the company's CRM, patients' codes, customers' names and prices that bind you. When a pilot goes wrong at this level, the damage lands outside your team. This book shows you how to set a pilot up so it answers its question without that happening, and how to end it with a decision.
After this you can
Write a one-page pilot plan with one question, a scope, named users, dates and exclusions.
Get data permissions in writing from the people who can give them, before the first real record.
Set success criteria, stop triggers and a decision rule before the pilot starts.
Keep a way back: human review before anything reaches a real system or person, and a tested undo.
Close the pilot with a results note and a go, change or stop decision someone can act on.
A candidate solution that passed a small experiment and a problem statement two colleagues signed.
The assistant your organisation approved for this kind of data: the institute's hosted assistant, your company's Microsoft Copilot or a managed workspace such as Langdock (a chat workspace a company buys for its staff, with admin controls), or the AI feature inside a system you already use. A personal account is never the right place for a pilot with real data.
A shared spreadsheet for the pilot log, in your team's usual place (Excel in SharePoint, Google Sheets). One row per case, filled in as you go. A spreadsheet beats a dashboard here: everyone can open it, and you can check any figure back to its row.
The people who can say yes: whoever owns the data and the system (the data owner, the system administrator), your data protection officer, and, in a company with a works council (in Germany the Betriebsrat, the elected body that represents employees), whoever talks to it.
About 4 hours of your time before and after, plus the pilot itself: typically four weeks.
In the four quadrants a pilot sits on the border. You've crossed into Choose tools and Build it far enough to have something that runs. But what the pilot produces is understanding: does this solve the problem you signed, for these people, on this data? So a pilot is Understand it at full scale, and it ends with a trip back to Structure it, where the decision gets written down.
Settle six things before day one. Once people depend on the pilot and data is flowing, you can't set these fairly any more.
Before day one
What to write down
1. The question
One sentence you'll answer with yes or no. "Does X let Y do Z in under N, without W?"
2. Scope
Named users, which work, which data, which systems, start and end dates, and what's excluded.
3. Data permissions
Which data, whose it is, who allowed its use in this tool, in writing.
4. Success criteria
Three or four measures, each with today's figure and a target. At least one measures harm.
5. Stop triggers and a way back
What pauses the pilot at once, who decides whether it restarts, and how you undo what it did.
6. The decision rule
Which results mean go, which mean change, which mean stop, and who decides.
Data permissions come from people, not from assumptions. "It's internal" doesn't answer the question. Ask three things, in writing. Does the data owner allow this use? Does the provider's contract cover this data? Look for a data processing agreement, the contract that says how the provider may handle your data, and for where the processing happens. Is any of it personal data, and if so, has your data protection officer seen the plan? When a permission is still pending, narrow the scope: run the pilot on the data you're cleared for and add the rest when the answer arrives.
Measure the work, not the tool. Logins, prompts sent and drafts generated tell you people clicked. Measure what the problem statement cares about: correct entries, replies sent in time, minutes saved, and one harm measure (errors that reached a real system or a real person). Measure the team, not named individuals. In Germany, where a company has a works council, introducing a technical system that can monitor employees' behaviour or performance needs its agreement. Talk to it before the pilot starts; measuring the team is usually all a pilot needs.
Always keep a way back. Nothing the AI produces reaches a real system or a real person without a human check. Before anything is written into a real system, take an export you know how to restore, and practise the restore once before day one.
Three decisions, all legitimate.Go: the criteria are met; the solution moves on to a proper build or rollout. Change: close, with a specific fix; run one more short round. Stop: it doesn't solve the problem, or it costs more than it saves. Record why. A stopped pilot with a clear reason is a finished pilot, and the reason will save the next person a month.
Worked example: a four-week pilot, start to decision
Both versions follow the same steps. Follow the one closer to your work. The plans, numbers and incidents are typical examples written for this book.
Lab: bench notes into the institute's inventory
The exploration note from T17-L03 · Explore What Already Exists Before You Build showed that finding samples is easy once they're recorded and labelled at the bench. The candidate solution: a colleague types or dictates a short note ("3 aliquots liver lysate mouse 212, rack A2 box 7, positions 1–3"), the institute's hosted assistant turns it into rows for the lab notebook's inventory import, and the colleague checks them and imports them.
Structure it: the pilot plan (90 min). The postdoc writes one page in the group's shared folder:
Question: Does the assistant let group members record new samples at the bench in under one minute each, with fields correct, so that samples are findable by others?
Scope: three users (the postdoc, one PhD student, one technician who was sceptical in the first conversation); racks A1 to A3 only; 4 weeks, 6 October to 31 October; new samples only, no old spreadsheet rows.
Excluded until cleared: samples from the clinical collaboration, which carry pseudonymised patient codes.
Systems: the institute's hosted assistant (approved for internal data); the lab notebook's inventory, written to only through its import, never by the assistant.
Success criteria (baseline from the exploration note):
Measure
Today
Target
Fields correct in a weekly spot check of 20 entries
not recorded
at least 98%
Median time from freezer to entry
entered days later, if at all
under 1 minute
New samples entered the same day
about 40%
at least 85%
Harm: a wrong location that sent someone to the wrong box, or a patient code outside an approved tool
not recorded
zero
Pause at once if: a patient code appears in any tool that isn't approved; or a wrong location sends someone to the wrong box twice. The PI then decides within two days whether to restart, and writes down why.
Way back: export the inventory before each import; restore tested once on 2 October.
Decision rule: all four criteria met means go (extend to the group). Accuracy between 95% and 98% with a clear cause means change (one two-week round with a specific fix). A second pause for the same cause, or accuracy below 95%, means stop.
Decides: the PI, on 3 November.
Understand it: data permissions (30 min, plus waiting). She sends three emails. The lab notebook's administrator confirms that imports through the standard route, a CSV file (a plain table any spreadsheet can save), are fine and that the export and restore work as she thinks. The institute's IT confirms the hosted assistant is approved for internal research data. The data protection officer replies that the clinical samples are covered by the collaboration agreement only for systems named in it, and the hosted assistant isn't one of them yet. He'll check. So the clinical samples stay out, as planned. The pilot starts on time with what's cleared.
Build it: the setup (60 min). She saves the instruction for the assistant with a picture of the rack and box grid in words ("positions 1–81, row by row, 1 is top left"), makes the pilot log (one row per sample: date, user, note, generated row, correct yes/no per field, minutes), and tapes a card to each freezer: the link to the approved assistant and "No clinical samples in the pilot."
Understand it: running and measuring (4 weeks). Every Friday she spot-checks 20 entries against the tubes in the freezer. The log after two weeks:
Measure
Target
Weeks 1–2
Fields correct
≥ 98%
96%
Median time to entry
< 1 min
50 seconds
Same-day entry
≥ 85%
88%
Harm
zero
one incident
Almost every error is in the position field: notes like "top left, next to Anna's" become a position number that's wrong. The incident is a patient code: in week 1 the technician, in a hurry, pasted a clinical sample list into his personal chatbot account. He reported it himself the same afternoon. That's a pause trigger, so the pilot stops that afternoon. The PI and the data protection officer look at it together: the list held six pseudonymised codes and no names. The data protection officer records it and decides what, if anything, has to be reported further. Two days later the PI restarts the pilot, clinical samples still excluded, and records why: the cause was a slow approved tool and a missing fallback, not the pilot's design. The card on the freezer gets a second line: "Slow tool? Write it on paper and enter it later."
At the planned checkpoint after week 2 she makes one change, logged with its date: the note must give the position as a number, and the assistant replies "position missing" instead of guessing. Weeks 3 to 4: fields correct 99.3%, median 45 seconds, same-day entry 91%, no new harm.
Structure it: results note and decision (60 min). One page: the question, each criterion with its result, the incident and what was done, the one change and its effect, what the three users said (the technician: "faster than the spreadsheet once I stopped writing 'top left'"), and what it cost: about 12 hours of the postdoc's time, no licence cost. By the rule, weeks 1 to 2 alone would have meant change, and the change round met every criterion. The PI signs go, with two conditions: clinical samples join only when the data protection officer confirms the tool, and the freezer card stays.
After T17-L03 · Explore What Already Exists Before You Build, requests arrive as tickets in the CRM with a named person each. The candidate: the CRM's built-in AI drafts a reply to each quote request, using the product catalogue and the current price list.
Structure it: the pilot plan (90 min).
Question: Do AI drafts cut the time to a first reply on quote requests to under three working hours, without a wrong price reaching a customer?
Scope: two sales colleagues; quote requests for lab consumables only (not instruments, not returns or complaints); 4 weeks; drafts only, a person reviews and sends every reply.
Data: customer names, email addresses and order history (personal data), and the price list (confidential).
Success criteria: median time to first reply from 7 working hours to under 3; zero wrong prices sent; both colleagues rate "saves me time" at least 4 out of 5; cost within the budget set for the pilot.
Stop at once if: a wrong price is sent; customer data appears outside the CRM.
Way back: switch the AI feature off for the pilot group; the ticket queue works as before.
Decision rule: all met means go; time met but price errors in drafts caught by review means change; a wrong price sent, or time not improved, means stop. Decides: the head of sales.
Understand it: data permissions. The CRM administrator confirms the AI feature is part of the contract the company already has; the data protection officer checks that the provider's data processing agreement covers it and where the processing happens, and approves it for the pilot. The head of sales informs the works council: the pilot measures the team's reply times, not individual people's.
Understand it: running and measuring. After four weeks, 212 quote requests. Median time to first reply: 2.5 working hours. Satisfaction: 4 and 5. No wrong price was sent. But the review found wrong prices in 6 of 212 drafts (3%). The AI had taken a price from an old quote in the email thread instead of the current list. Checking every price against the list took the two colleagues longer than anything else.
Structure it: results note and decision. The time criterion is met; price errors appeared in drafts and were caught. By the rule, that's change. The specific change: the AI drafts the text, and prices go in through the CRM's price field, taken from the price list, never from the AI. A two-week second round: 104 requests, no price errors in drafts, median 2.2 hours, and the review got shorter. The head of sales signs go.
In both cases the first round didn't pass, and the pilot did its job: it showed exactly which part to change, and the rule said what to do about it before anyone could argue. And in both cases the harm criterion, not the time saved, was the one that shaped the final solution.
The results note (one page)
Question and dates. Copied from the plan.
What ran. Users, volume of work, any change made, with its date.
Results against each criterion. Target, result, and the log rows behind it.
Harm and incidents. What happened, who was told, what was done.
What users said. Two or three quotes, including the sceptic's.
Cost. Your hours, their hours, licences.
Decision. Go, change or stop, which rule it follows, conditions, next step, and the signature of the person who decides.
What goes wrong
The criteria are written after the results.Symptom: "we saw a 40% improvement" in a measure nobody had named before the pilot. Fix: date the criteria and the decision rule in the plan before day one and have the person who decides sign them. Anything you notice later goes in the note as an observation, not a criterion.
Only enthusiasts take part.Symptom: glowing results that fall apart when the rest of the team starts. Fix: include at least one typical user and one sceptic. Their numbers are the ones that predict the rollout.
Permission is assumed.Symptom: "it's all internal" until the data protection officer hears about it from someone else. Fix: ask the data owner, the system administrator and the data protection officer in writing before the first real record. Leave out anything not yet cleared.
No way back.Symptom: the AI writes straight into the real system, and nobody knows how to undo a bad batch. Fix: keep a person between the AI and every real system or person, take an export before each write, and practise the restore once before you start.
Measuring clicks.Symptom: the results note reports drafts generated and logins per week. Fix: measure the outcomes in your problem statement and one harm measure. If you can't count it from the log, it isn't a criterion.
Silent changes mid-pilot.Symptom: the prompt was "improved" in week 2 and nobody knows which results came before and after. Fix: plan one checkpoint. Log every change with its date, and report results before and after it separately.
The pilot never ends.Symptom: three months later it's "still in pilot", people depend on it, and nobody is responsible for it. Fix: put the end date and the decision meeting in the calendar on day one. On that date, one of the three decisions gets made, even if it's "change, two more weeks".
Do it yourself (4 hours plus the pilot)
Structure it: the plan (90 min). Write the one-page plan from your exploration note: question, scope, data, criteria with baselines, stop triggers, way back, decision rule, who decides. Get the decider to sign it.
Understand it: permissions (30 min, plus waiting). Email the data owner or system administrator, IT and your data protection officer, and, in a company with a works council, whoever speaks to it. Narrow the scope to what's cleared.
Build it: setup (60 min). Set up the tool for the pilot users only. Make the pilot log. Test the way back once: export, restore, check.
Understand it: run (the pilot period, 15 min a week). Fill the log as you go. Do the spot check every week. Hold the checkpoint. Log every change and incident with its date.
Structure it: close (60 min). Write the results note against every criterion, apply the decision rule, and get the decision signed on the planned date.
Exit check
Written finding: a pilot plan and a results note for a real pilot.
It passes when:
the question, criteria, stop triggers and decision rule are dated before the pilot started;
data permissions are confirmed in writing by the people who can give them;
every criterion, including at least one harm measure, has a measured result traceable to the log;
the go, change or stop decision follows the rule, or says why it doesn't, and is signed by the person who decides.
Rule to remember
Decide what counts as success, and what makes you stop, before the first real record goes in.
Langdock, Microsoft Copilot: Catalogued. Examples of managed assistants an organisation can approve for internal data.
Watch
Strategyzer Webinar: The Basics Of Testing Business Ideas
Strategyzer Webinar: The Basics Of Testing Business IdeasStrategyzer
T17-L04: Learn to choose a hypothesis, design a test and learn from its results. This 2018 business-experimentation webinar complements the pilot method; retain the book's data permissions, stop triggers and rollback controls.
