Keep a Data Pipeline Right for Years, and Trace Any Published Number to Its Source | Heidelberg AI Curriculum
T10-L05
Keep a Data Pipeline Right for Years, and Trace Any Published Number to Its Source
Run a data pipeline that many people rely on: keep every run's input, code and output under one run ID, watch that it ran and that its data still look like last month's, approve before anything is published, and trace any number in a board pack or paper back to its source rows, even after you've left.
In March the board pack shows travel costs in the sales cost centre up 180% on the previous year. The sales director spends a meeting defending a number that turns out to be an accounting change: in February, accounting moved car leasing into the travel cost type. Every check passed. The column names were right, the values in range, the data fresh. The data had simply stopped meaning what they meant a year earlier.
Then the auditor, preparing the annual audit, points at one cell in last year's cost analysis and asks where it came from. The report file was overwritten by every later run. The ERP data has changed since. Nobody can show which rows produced that number.
In a lab it's Clara's HPLC summary, which now feeds figures in two papers. A reviewer asks for the raw values behind one bar in a figure from 18 months ago. Clara is leaving in the summer.
You are at Level 5 Operator. Many people rely on this pipeline, and some of its numbers leave the organisation in board packs, audits and papers. When it's wrong, decisions go wrong across the organisation. This book shows you how to watch it, keep what you'll need to prove a number, and hand it over so it keeps working without you.
After this you can
Keep every run's source extract, code version, check results and output together under one run ID, and never overwrite them.
Watch that the pipeline ran on time and that its data still look like the approved baseline, with alerts to named people.
Separate the person who prepares a result from the person who approves it for publication.
Trace one published number back to its source rows, with the retention rule for each piece, in a way a successor can repeat without you.
A pipeline that already publishes, with its code in Git and a named deputy.
Access to your IT's monitoring tool, or to an Uptime Kumaserver. Uptime Kuma is a free monitoring program many IT teams run. Ask IT which they use.
The retention rules that apply: your finance department's rules for accounting records, or your institution's rules for research data (often ten years from publication).
About 4¼ hours: 17 minutes to read, about 4 hours of doing, over three weeks.
Last verified 23 September 2026. Uptime Kuma is free and open source; its current version is 2.5.5. It can raise an alarm when a scheduled job doesn't report in. Details for experts at the end of this book.
The idea in one page
A pipeline that runs every week for years fails in four ways that no single run shows you. Each has an answer.
It fails because…
What you see
The answer
It didn't run
Last week's result, still there, looking current
An alarm outside the pipeline that fires when no run reports in
The data changed meaning
Every check green, a number that makes no sense
A comparison with an approved baseline: does this month look like the months before?
Nobody can prove a number
"It came from the pipeline", and nothing more
A sealed run folder for every run, and a trace from the number back to its rows
Only you can run it
Work stops when you're away
A runbook, a deputy who has done it, accounts that belong to the organisation
A run ID for every run, and nothing ever overwritten. Every run writes one folder, named by date and a running number, with five things: the rows extracted from the source, the code version (the Git commit), the settings, the check results, and the output. Each file gets a fingerprint, the short code from T10-L03 · Build an Analysis Your Colleague Can Rerun and Get the Same Numbers that changes if any byte changes. A number published from run 0042 can always be found in folder 0042. Mark the folders read-only once written, and let IT back them up.
A pipeline can't report that it never ran. So the alarm lives outside it. At the end of a successful run, the pipeline calls a web address on the monitoring server. If no call arrives by Monday 07:30, the monitoring server alerts you and your deputy. This is called a heartbeat: silence is the signal.
Format checks stop; drift checks warn.T10-L04 · Connect Your Analysis to a Live System, and Stop Bad Data Before It Reaches Anyone's checks catch a wrong shape: they stop the run. Drift is different: the data are well formed but have shifted, like leasing costs appearing in travel. Compare each run with a baseline, the months someone approved as normal. Flag any cost centre or group whose total moves more than an agreed amount against the same month last year, or against the average of the last twelve months, and any segment that is suddenly empty. A drift flag doesn't stop the run. It holds the result for a person to look at before it's published.
The person who prepares isn't the person who approves. The pipeline and its operator prepare a result. Someone else, the controller or the PI (the group's lead researcher), approves that exact run for a named place: this run, this number, this slide. The approval goes into the run folder. That separation is what an auditor looks for, and it catches the mistakes the operator can't see.
Keep each piece as long as it's needed, and no longer. Source extracts, outputs and approvals for anything that was published are kept as long as the retention rule says. Runs that were stopped or never published can go after a few months. Personal data should never be in the run folders in the first place, because T10-L04 · Connect Your Analysis to a Live System, and Stop Bad Data Before It Reaches Anyone's view leaves it out.
What you watch, what you do, who takes over, what you show. You watch the heartbeat, the drift flags and a monthly glance at the run history. When it breaks, the runbook says who does what. The deputy has run it. And for any published number, you can show the trace.
Worked example: from "it came from the pipeline" to a trace, four quadrants
Structure it. After the March meeting and the auditor's question, Deniz writes the problem down with the controller. "Any number the pipeline puts into a management or board document can be traced back to its source rows for as long as accounting records are kept. A number that looks normal but means something new is caught before the board sees it. The pipeline keeps running when I'm away, and we know within half an hour if it didn't."
Understand it. She starts by trying to trace the auditor's number herself, and writes down every place the trail goes cold.
The report for that month was overwritten every Monday until the month closed. The final version exists only as a PDF in the board pack.
The code has changed three times since. Git has the history, but nothing records which commit produced which report.
The ERP's reporting view can be queried for last year, but accounting has since corrected four bookings in that month, so today's query gives a different sum.
Two Mondays in the last six months, the job didn't run at all: once the server was being patched, once the service account's password had expired. Both times people read the previous week's report without noticing.
Then she looks at the March case. She compares the travel cost type over the last 24 months and sees the jump in February, in every cost centre with company cars. A simple rule would have caught it: more than 50% above the same month last year, in more than one cost centre at once.
The finding that changes her problem is the corrected bookings. She had planned to prove numbers by rerunning the pipeline on old data. But the source changes after the fact, legitimately, so a rerun proves nothing. Back to Structure it: "Each run keeps its own copy of what it read. The trace goes through that copy, never through today's database."
Choose tools.
Run folders on the finance file server, which IT backs up and where retention is already managed. After each run a small step marks the folder read-only. No new system.
Uptime Kuma, which IT already runs for its servers, with a push monitor: a monitor that waits for the pipeline to call a web address and alerts if the call doesn't come. It fits because it's already there and has email alerts. The alternative is Healthchecks.io, a hosted service built only for this job.
The drift check in pandas, in the pipeline itself. It's twenty lines, and the controller can read the rule.
Approval as a file in the run folder, written by a small script the controller runs, rather than an email that gets lost.
Build it: the run folder. Every run now writes:
runs/2026-10-05_0042/
source.parquet what the view returned, as read
settings.toml the month, the budget version, the thresholds
checks.json every check and drift result, with the numbers
report.md the output
run_info.json run ID, Git commit, fingerprint of uv.lock (the package
versions), start and end time, who started it, the
operator, fingerprint of every file above
approval.json added later by the approver
The source rows are stored as Parquet, a compact file format for tables that keeps each column's type exactly. pandas needs one more library to write it, added with uv add pyarrow. The extract is small (a few hundred summed rows, no names), so keeping every run costs almost nothing. The last line of the pipeline sets the files to read-only, and IT removes the service account's right to write into finished run folders, which is the lock that really holds.
Build it: the heartbeat. IT creates a push monitor in Uptime Kuma named Cost review, weekly, alerting Deniz, her deputy Ole and the controller by email. Its heartbeat interval, in seconds, is set to seven days plus 90 minutes, 610,200 seconds, so a run that normally reports around 06:00 is missed by 07:30. Uptime Kuma shows a secret web address, the push URL; IT puts it into the job's environment next to the database password. After a successful run the pipeline calls it with Python's built-in urllib:
This one line tells the monitor that run 0042 finished; %20 is how a web address writes a space. A stopped run doesn't call it, so the stop email from T10-L04 · Connect Your Analysis to a Live System, and Stop Bad Data Before It Reaches Anyone and the missing heartbeat both reach the team. To test it, IT pauses the job for one week. At 07:30 on Monday the alert arrives.
Build it: the drift check. Deniz writes the rules with the controller and puts them in the settings file, where the controller can read and sign them:
[drift]
baseline = "same month last year, and mean of last 12 approved months"
warn_if_change_over = 0.5 # 50% up or down against either baseline
warn_if_cost_centres_affected = 2
warn_if_segment_empty = true # a cost centre with postings last year, none now
approved_by = "Controller, 2026-10-02"
The drift check reads the last twelve approved run folders, not the database, so the baseline is what was actually published. It writes every comparison into checks.json. When a rule fires, the report is marked HELD: drift at the top, and the controller gets a list of the flagged lines. A held report can't be approved without a written explanation, which the controller gives when she approves.
Build it: prepare, then approve. A report isn't published because the run finished. The controller looks at it, and if she accepts it for a named place, she runs one command that writes approval.json into the run folder:
uv run --locked approve.py runs/2026-10-05_0042 --for "Board pack October, slide 4" --explain "leasing moved into travel from Feb 2026, see accounting memo 2026-02"
The script, which Deniz drafts with OpenCode and the controller reads, is short:
"""Approve one run for one named place. Refuses self-approval and changed reports."""
import argparse, datetime as dt, getpass, hashlib, json, pathlib
a = argparse.ArgumentParser()
a.add_argument("run"); a.add_argument("--for", dest="where", required=True)
a.add_argument("--explain", default="")
args = a.parse_args()
run = pathlib.Path(args.run)
info = json.loads((run / "run_info.json").read_text(encoding="utf-8"))
checks = json.loads((run / "checks.json").read_text(encoding="utf-8"))
me = getpass.getuser() # the Windows login of whoever runs this
report = hashlib.blake2b((run / "report.md").read_bytes(), digest_size=16).hexdigest()
if report != info["fingerprints"]["report.md"]:
raise SystemExit("REFUSED: report.md has changed since the run.")
if me in (info["started_by"], info["operator"]):
raise SystemExit(f"REFUSED: {me} prepared this run and can't approve it.")
if any(flag["fired"] for flag in checks["drift"]) and not args.explain:
raise SystemExit("REFUSED: drift flags fired; add --explain with the reason.")
(run / "approval.json").write_text(json.dumps({
"approved_by": me, "date": dt.date.today().isoformat(), "for": args.where,
"report_fingerprint": report, "drift_explanation": args.explain}, indent=2), encoding="utf-8")
print(f"Approved {run.name} for {args.where}")
It records her login name, the date, the destination, her explanation and the fingerprint of report.md as it was when she approved it. Deniz, named as operator in the settings, can't approve her own runs, and neither can the service account that started them.
Understand it: trace the auditor's number. With the new setup running, Deniz runs a drill. She picks a number from the October board pack, travel costs for cost centre 3100, €84,210, and traces it from the slide back to the rows, writing each step on one page.
Where it appears: board pack October, slide 4, cell Travel, 3100.
Approval: The approval file in run 0042 names that slide, the controller, the date, and the report's fingerprint.
Output: The report in run 0042, whose fingerprint matches the approval, shows €84,210.
Code: The run information names Git commit a41f9c2; the code at that commit sums the source rows per cost centre and type.
Source rows: The source extract, whose fingerprint matches the run information, holds three rows for 3100 and travel: September, 84,210.00 from 212 postings. The view's definition at that date is in the field list, version 2, signed by the head of finance.
Then her deputy repeats it with a different number, using only the runbook. He gets stuck once: the runbook says "check the fingerprint" but not how. She adds the one command that computes it, the same kind of fingerprint as in T10-L03 · Build an Analysis Your Colleague Can Rerun and Get the Same Numbers, for any file:
uv run python -c "import hashlib,sys; print(hashlib.blake2b(open(sys.argv[1],'rb').read(), digest_size=16).hexdigest())" runs/2026-10-05_0042/source.parquet
The printed code must match the one in run_info.json.
Build it: retention and the runbook. The retention plan has one line per kind of file, agreed with the head of accounting:
run folders whose output was approved for a board or annual document: kept as long as accounting records;
runs that were stopped, or held and never approved: deleted after six months;
the monitoring history: kept in Uptime Kuma for a year.
The runbook adds the new parts to T10-L04 · Connect Your Analysis to a Live System, and Stop Bad Data Before It Reaches Anyone's run sheet: what a missing heartbeat means, how to read a drift flag, and how to trace a number. It also says who approves when the controller is away (the head of accounting), and how to renew the service account's password before it expires, with a calendar reminder owned by IT.
For the old auditor's number, there's no run folder, and Deniz says so plainly. She gives the auditor the board pack PDF, the Git history for that month and the list of four later corrections from accounting. The auditor accepts the explanation and notes that the new process covers the current year.
What it taught her: the heartbeat and the run folders were an afternoon. Learning that a rerun on today's data proves nothing about last year's number took a failed audit question, and it changed everything else.
Lab: a figure from 18 months ago
Structure it. Clara leaves in the summer. "Any value in a published figure can be traced to the LIMS rows it came from, by Ben, for ten years."
Understand it. She tries the reviewer's request. The figure's commit and settings are in Git, from T10-L03 · Build an Analysis Your Colleague Can Rerun and Get the Same Numbers. But the LIMS results for that month were re-integrated after a peak-picking fix, and the view now returns slightly different values. The same finding as in the company: the source changes afterwards, legitimately. Back to Structure it: each weekly run keeps what it read.
Choose tools and Build it. The same run folders, on the institute's research data storage, where the ten-year retention rule of the German Research Foundation (DFG) is already handled. A drift rule agreed with the PI: flag a group whose mean moves more than three standard deviations from the last twelve approved weeks, and any instrument with no results this week. A heartbeat on the institute's Uptime Kuma. The PI approves a run for a named figure before it goes into a manuscript, and the approval names the figure. Ben does the trace drill on the figure from the reviewer's request, from the runbook alone. For that one, Clara can give the reviewer the committed settings and the LIMS audit trail, the LIMS's own record of every change to a result, which shows the re-integration; the new runs won't need that detour.
In both cases, what made a number provable wasn't rerunning the code. It was keeping, at the time, exactly what the code had read, and a person's approval for exactly that output.
What goes wrong
Proving a number by rerunning it.Symptom: the rerun gives a different result because the source has been corrected since. Fix: each run keeps its own extract; the trace goes through the run folder, never through today's database.
Silence taken for success.Symptom: last week's report is still there, and nobody notices the job didn't run. Fix: a heartbeat to a monitor outside the pipeline, with alerts to two named people, tested once by pausing the job.
Every check green, the number meaningless.Symptom: a cost type or unit quietly changes what it contains. Fix: a drift check against approved runs, agreed with the person who knows the data, that holds the result for a person rather than stopping it.
The operator approves their own run.Symptom: one person prepares, checks and publishes. Fix: an approval file written by someone else, bound to the report's fingerprint and to a named destination, and a script that refuses self-approval.
Everything kept forever.Symptom: thousands of run folders, including stopped test runs, with no deletion date. Fix: a retention line per kind of file, a person who deletes, and personal data kept out of the extracts from the start.
An account that expires.Symptom: the job stops on a Monday because the service account's password ran out. Fix: the expiry date in IT's calendar with a named person responsible, and the heartbeat to catch it if the reminder fails.
The runbook nobody has used.Symptom: the deputy opens it for the first time during an incident. Fix: a trace drill and a missed-run drill, done by the deputy, once a quarter, with the runbook fixed each time.
Do it yourself (4 hours, over three weeks)
Structure it (15 min). Write which numbers from your pipeline leave your team, where they go, how long they must be provable and who approves them.
Understand it (45 min). Try to trace one published number from last year. Write down every place the trail goes cold. Look at 24 months of your data for a change in meaning that passed every check. Rewrite your problem sentence.
Choose tools (15 min). Decide where run folders live and how they're backed up. Ask IT for a push monitor. Agree the drift rules and who approves.
Build it: run folders and heartbeat (60 min). Make every run write its folder with fingerprints, mark it read-only, and call the heartbeat at the end. Test the alert by pausing the job.
Build it: drift and approval (60 min). Write the drift check from approved runs, and the approval script that refuses self-approval and changed reports. Test both with a planted shift in a copy of the data.
Understand it: trace drill (30 min). Trace one current published number back to its rows on one page. Have your deputy trace a second one from the runbook alone.
Build it: retention and runbook (15 min). Write one retention line per kind of file with the person who deletes. Put the next quarterly drill in a shared calendar.
Exit check
Written finding: a trace of one number that was published from the pipeline, with:
where it appears, and the approval that names that place;
the run ID, and the fingerprints of the output, the source extract and the code version;
the check and drift results for that run, with any explanation;
the source rows that produced the number, and the field list that defined the view;
the retention rule for each piece, and who deletes it.
It passes when every step names its run ID, fingerprint or person, the missed-run alert and the drift check have each fired once in a test, and a colleague repeats the trace from the runbook alone and reaches the same rows.
Rule to remember
Keep what each run read, and you can prove any number it published.
Details for experts
Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.
What
Details
What it means for you
Uptime Kuma
Version 2.5.5, released 16 September 2026, MIT licence, self-hosted.
Ask IT to use their instance before setting up your own.
Push monitor
The job calls /api/push/<token> on the Uptime Kuma server; optional status (up or down) and msg parameters. No call within the heartbeat interval turns the monitor red and sends the alert.
The token in the address is a secret: keep it in the job's environment, not in Git.
Parquet
A column-based file format for tables. pandas writes it with to_parquet when the pyarrow library is installed.
Already added in the run-folder step.
Read-only folders
On a Windows file server, IT can remove write permission for the service account after each run, or you can set the read-only attribute from the script. Only the first stops a determined writer.
Ask IT for the permission route for published runs.
DFG
Guideline 17 of the DFG's Code of Conduct: research data and results are archived, as a rule, for ten years.
The lab's run folders belong on storage with that retention.
Uptime Kuma: Taught. The push monitor that notices a run that didn't happen.
Git: Taught. Tags that name the code behind every run.
KNIME: Catalogued. A visual alternative with its own run history.
Watch
How to Set Up Heartbeat Monitoring for Cron Jobs
How to Set Up Heartbeat Monitoring for Cron JobsPulsetic
T05-L05: Learn why an external monitor can detect an automation that never ran. Start at 0:13; send the ping only after successful completion. This Pulsetic example teaches the same pattern as the book's Kuma Push monitor, not n8n wiring. Use your actual schedule and keep ping URLs private.
T10-L05: See completion-based heartbeat monitoring with intervals, grace periods and missing-ping alerts. Put the ping after the pipeline succeeds and use its real deadline. This Pulsetic demonstration complements the book's external alarm, not its run provenance, drift checks or handover.
