Keep a Data Pipeline Right for Years, and Trace Any Published Number to Its Source

T10-L05 · Data & analysis · Level 5 Operator · 17-minute read, plus 4 hours of doing

A green run and a number nobody can explain

Six months after Deniz connected the cost review to the ERP (T10-L04), four things depend on it. The Monday management meeting, the monthly board pack, the plant managers' budget calls, and the annual report's cost analysis. Every run has been green.

In March the board pack shows travel costs in the sales cost centre up 180% on the previous year. The sales director spends a meeting defending a number that turns out to be an accounting change: in February, accounting moved car leasing into the travel cost type. Every check passed. The column names were right, the values in range, the data fresh. The data had simply stopped meaning what they meant a year earlier.

Then the auditor, preparing the annual audit, points at one cell in last year's cost analysis and asks where it came from. The report file was overwritten by every later run. The ERP data has changed since. Nobody can show which rows produced that number.

In a lab it's Clara's HPLC summary, which now feeds figures in two papers. A reviewer asks for the raw values behind one bar in a figure from 18 months ago. Clara is leaving in the summer.

You are at Level 5 Operator. Many people rely on this pipeline, and some of its numbers leave the organisation in board packs, audits and papers. When it's wrong, decisions go wrong across the organisation. This book shows you how to watch it, keep what you'll need to prove a number, and hand it over so it keeps working without you.

After this you can

• Keep every run's source extract, code version, check results and output together under one run ID, and never overwrite them.
• Watch that the pipeline ran on time and that its data still look like the approved baseline, with alerts to named people.
• Separate the person who prepares a result from the person who approves it for publication.
• Trace one published number back to its source rows, with the retention rule for each piece, in a way a successor can repeat without you.

Before you start

• Books: T10-L04 for the live pipeline with its view, checks and run sheet. T12-L05 for the organisation-wide inventory this pipeline belongs in. The four-quadrant method is in T01-L01.
• A pipeline that already publishes, with its code in Git and a named deputy.
• Access to your IT's monitoring tool, or to an Uptime Kuma server. Uptime Kuma is a free monitoring program many IT teams run. Ask IT which they use.
• The retention rules that apply: your finance department's rules for accounting records, or your institution's rules for research data (often ten years from publication).
• About 4¼ hours: 17 minutes to read, about 4 hours of doing, over three weeks.

Last verified 23 September 2026. Uptime Kuma is free and open source; its current version is 2.5.5. It can raise an alarm when a scheduled job doesn't report in. Details for experts at the end of this book.

The idea in one page

A pipeline that runs every week for years fails in four ways that no single run shows you. Each has an answer.

A run ID for every run, and nothing ever overwritten. Every run writes one folder, named by date and a running number, with five things: the rows extracted from the source, the code version (the Git commit), the settings, the check results, and the output. Each file gets a fingerprint, the short code from T10-L03 that changes if any byte changes. A number published from run 0042 can always be found in folder 0042. Mark the folders read-only once written, and let IT back them up.

A pipeline can't report that it never ran. So the alarm lives outside it. At the end of a successful run, the pipeline calls a web address on the monitoring server. If no call arrives by Monday 07:30, the monitoring server alerts you and your deputy. This is called a heartbeat: silence is the signal.

Format checks stop; drift checks warn. T10-L04's checks catch a wrong shape: they stop the run. Drift is different: the data are well formed but have shifted, like leasing costs appearing in travel. Compare each run with a baseline, the months someone approved as normal. Flag any cost centre or group whose total moves more than an agreed amount against the same month last year, or against the average of the last twelve months, and any segment that is suddenly empty. A drift flag doesn't stop the run. It holds the result for a person to look at before it's published.

The person who prepares isn't the person who approves. The pipeline and its operator prepare a result. Someone else, the controller or the PI (the group's lead researcher), approves that exact run for a named place: this run, this number, this slide. The approval goes into the run folder. That separation is what an auditor looks for, and it catches the mistakes the operator can't see.

Keep each piece as long as it's needed, and no longer. Source extracts, outputs and approvals for anything that was published are kept as long as the retention rule says. Runs that were stopped or never published can go after a few months. Personal data should never be in the run folders in the first place, because T10-L04's view leaves it out.

What you watch, what you do, who takes over, what you show. You watch the heartbeat, the drift flags and a monthly glance at the run history. When it breaks, the runbook says who does what. The deputy has run it. And for any published number, you can show the trace.

Worked example: from "it came from the pipeline" to a trace, four quadrants

Both versions follow the four quadrants from T01-L01. The company one is told in full, continuing T10-L04. Names, numbers and dates are illustrations.

Company: the cost review, six months on

Structure it. After the March meeting and the auditor's question, Deniz writes the problem down with the controller. "Any number the pipeline puts into a management or board document can be traced back to its source rows for as long as accounting records are kept. A number that looks normal but means something new is caught before the board sees it. The pipeline keeps running when I'm away, and we know within half an hour if it didn't."

Understand it. She starts by trying to trace the auditor's number herself, and writes down every place the trail goes cold.

• The report for that month was overwritten every Monday until the month closed. The final version exists only as a PDF in the board pack.
• The code has changed three times since. Git has the history, but nothing records which commit produced which report.
• The ERP's reporting view can be queried for last year, but accounting has since corrected four bookings in that month, so today's query gives a different sum.
• Two Mondays in the last six months, the job didn't run at all: once the server was being patched, once the service account's password had expired. Both times people read the previous week's report without noticing.

Then she looks at the March case. She compares the travel cost type over the last 24 months and sees the jump in February, in every cost centre with company cars. A simple rule would have caught it: more than 50% above the same month last year, in more than one cost centre at once.

The finding that changes her problem is the corrected bookings. She had planned to prove numbers by rerunning the pipeline on old data. But the source changes after the fact, legitimately, so a rerun proves nothing. Back to Structure it: "Each run keeps its own copy of what it read. The trace goes through that copy, never through today's database."

Choose tools.

• Run folders on the finance file server, which IT backs up and where retention is already managed. After each run a small step marks the folder read-only. No new system.
• Git tags, names attached to a commit such as release-2026-10, for code versions, so a run folder names the exact code, as in T10-L03.
• Uptime Kuma, which IT already runs for its servers, with a push monitor: a monitor that waits for the pipeline to call a web address and alerts if the call doesn't come. It fits because it's already there and has email alerts. The alternative is Healthchecks.io, a hosted service built only for this job.
• The drift check in pandas, in the pipeline itself. It's twenty lines, and the controller can read the rule.
• Approval as a file in the run folder, written by a small script the controller runs, rather than an email that gets lost.

Build it: the run folder. Every run now writes:

runs/2026-10-05_0042/
  source.parquet        what the view returned, as read
  settings.toml         the month, the budget version, the thresholds
  checks.json           every check and drift result, with the numbers
  report.md             the output
  run_info.json         run ID, Git commit, fingerprint of uv.lock (the package
                        versions), start and end time, who started it, the
                        operator, fingerprint of every file above
  approval.json         added later by the approver

The source rows are stored as Parquet, a compact file format for tables that keeps each column's type exactly. pandas needs one more library to write it, added with uv add pyarrow. The extract is small (a few hundred summed rows, no names), so keeping every run costs almost nothing. The last line of the pipeline sets the files to read-only, and IT removes the service account's right to write into finished run folders, which is the lock that really holds.

Build it: the heartbeat. IT creates a push monitor in Uptime Kuma named Cost review, weekly, alerting Deniz, her deputy Ole and the controller by email. Its heartbeat interval, in seconds, is set to seven days plus 90 minutes, 610,200 seconds, so a run that normally reports around 06:00 is missed by 07:30. Uptime Kuma shows a secret web address, the push URL; IT puts it into the job's environment next to the database password. After a successful run the pipeline calls it with Python's built-in urllib:

urllib.request.urlopen(os.environ["HEARTBEAT_URL"] + "?status=up&msg=run%200042", timeout=10)

This one line tells the monitor that run 0042 finished; %20 is how a web address writes a space. A stopped run doesn't call it, so the stop email from T10-L04 and the missing heartbeat both reach the team. To test it, IT pauses the job for one week. At 07:30 on Monday the alert arrives.

Build it: the drift check. Deniz writes the rules with the controller and puts them in the settings file, where the controller can read and sign them:

[drift]
baseline = "same month last year, and mean of last 12 approved months"
warn_if_change_over = 0.5        # 50% up or down against either baseline
warn_if_cost_centres_affected = 2
warn_if_segment_empty = true     # a cost centre with postings last year, none now
approved_by = "Controller, 2026-10-02"

The drift check reads the last twelve approved run folders, not the database, so the baseline is what was actually published. It writes every comparison into checks.json. When a rule fires, the report is marked HELD: drift at the top, and the controller gets a list of the flagged lines. A held report can't be approved without a written explanation, which the controller gives when she approves.

Build it: prepare, then approve. A report isn't published because the run finished. The controller looks at it, and if she accepts it for a named place, she runs one command that writes approval.json into the run folder:

uv run --locked approve.py runs/2026-10-05_0042 --for "Board pack October, slide 4" --explain "leasing moved into travel from Feb 2026, see accounting memo 2026-02"

The script, which Deniz drafts with OpenCode and the controller reads, is short:

"""Approve one run for one named place. Refuses self-approval and changed reports."""
import argparse, datetime as dt, getpass, hashlib, json, pathlib

a = argparse.ArgumentParser()
a.add_argument("run"); a.add_argument("--for", dest="where", required=True)
a.add_argument("--explain", default="")
args = a.parse_args()
run = pathlib.Path(args.run)
info = json.loads((run / "run_info.json").read_text(encoding="utf-8"))
checks = json.loads((run / "checks.json").read_text(encoding="utf-8"))
me = getpass.getuser()                       # the Windows login of whoever runs this
report = hashlib.blake2b((run / "report.md").read_bytes(), digest_size=16).hexdigest()

if report != info["fingerprints"]["report.md"]:
    raise SystemExit("REFUSED: report.md has changed since the run.")
if me in (info["started_by"], info["operator"]):
    raise SystemExit(f"REFUSED: {me} prepared this run and can't approve it.")
if any(flag["fired"] for flag in checks["drift"]) and not args.explain:
    raise SystemExit("REFUSED: drift flags fired; add --explain with the reason.")
(run / "approval.json").write_text(json.dumps({
    "approved_by": me, "date": dt.date.today().isoformat(), "for": args.where,
    "report_fingerprint": report, "drift_explanation": args.explain}, indent=2), encoding="utf-8")
print(f"Approved {run.name} for {args.where}")

It records her login name, the date, the destination, her explanation and the fingerprint of report.md as it was when she approved it. Deniz, named as operator in the settings, can't approve her own runs, and neither can the service account that started them.

Understand it: trace the auditor's number. With the new setup running, Deniz runs a drill. She picks a number from the October board pack, travel costs for cost centre 3100, €84,210, and traces it from the slide back to the rows, writing each step on one page.

1. Where it appears: board pack October, slide 4, cell Travel, 3100.
2. Approval: The approval file in run 0042 names that slide, the controller, the date, and the report's fingerprint.
3. Output: The report in run 0042, whose fingerprint matches the approval, shows €84,210.
4. Checks: The checks file shows all T10-L04 checks passed and one drift flag on 3100; the approval carries the controller's explanation: "leasing moved into travel from Feb 2026, see accounting memo 2026-02".
5. Code: The run information names Git commit a41f9c2; the code at that commit sums the source rows per cost centre and type.
6. Source rows: The source extract, whose fingerprint matches the run information, holds three rows for 3100 and travel: September, 84,210.00 from 212 postings. The view's definition at that date is in the field list, version 2, signed by the head of finance.

Then her deputy repeats it with a different number, using only the runbook. He gets stuck once: the runbook says "check the fingerprint" but not how. She adds the one command that computes it, the same kind of fingerprint as in T10-L03, for any file:

uv run python -c "import hashlib,sys; print(hashlib.blake2b(open(sys.argv[1],'rb').read(), digest_size=16).hexdigest())" runs/2026-10-05_0042/source.parquet

The printed code must match the one in run_info.json.

Build it: retention and the runbook. The retention plan has one line per kind of file, agreed with the head of accounting:

• run folders whose output was approved for a board or annual document: kept as long as accounting records;
• runs that were stopped, or held and never approved: deleted after six months;
• the monitoring history: kept in Uptime Kuma for a year.

The runbook adds the new parts to T10-L04's run sheet: what a missing heartbeat means, how to read a drift flag, and how to trace a number. It also says who approves when the controller is away (the head of accounting), and how to renew the service account's password before it expires, with a calendar reminder owned by IT.

For the old auditor's number, there's no run folder, and Deniz says so plainly. She gives the auditor the board pack PDF, the Git history for that month and the list of four later corrections from accounting. The auditor accepts the explanation and notes that the new process covers the current year.

What it taught her: the heartbeat and the run folders were an afternoon. Learning that a rerun on today's data proves nothing about last year's number took a failed audit question, and it changed everything else.

Lab: a figure from 18 months ago

Structure it. Clara leaves in the summer. "Any value in a published figure can be traced to the LIMS rows it came from, by Ben, for ten years."

Understand it. She tries the reviewer's request. The figure's commit and settings are in Git, from T10-L03. But the LIMS results for that month were re-integrated after a peak-picking fix, and the view now returns slightly different values. The same finding as in the company: the source changes afterwards, legitimately. Back to Structure it: each weekly run keeps what it read.

Choose tools and Build it. The same run folders, on the institute's research data storage, where the ten-year retention rule of the German Research Foundation (DFG) is already handled. A drift rule agreed with the PI: flag a group whose mean moves more than three standard deviations from the last twelve approved weeks, and any instrument with no results this week. A heartbeat on the institute's Uptime Kuma. The PI approves a run for a named figure before it goes into a manuscript, and the approval names the figure. Ben does the trace drill on the figure from the reviewer's request, from the runbook alone. For that one, Clara can give the reviewer the committed settings and the LIMS audit trail, the LIMS's own record of every change to a result, which shows the re-integration; the new runs won't need that detour.

In both cases, what made a number provable wasn't rerunning the code. It was keeping, at the time, exactly what the code had read, and a person's approval for exactly that output.

What goes wrong

Proving a number by rerunning it. Symptom: the rerun gives a different result because the source has been corrected since. Fix: each run keeps its own extract; the trace goes through the run folder, never through today's database.

Silence taken for success. Symptom: last week's report is still there, and nobody notices the job didn't run. Fix: a heartbeat to a monitor outside the pipeline, with alerts to two named people, tested once by pausing the job.

Every check green, the number meaningless. Symptom: a cost type or unit quietly changes what it contains. Fix: a drift check against approved runs, agreed with the person who knows the data, that holds the result for a person rather than stopping it.

The operator approves their own run. Symptom: one person prepares, checks and publishes. Fix: an approval file written by someone else, bound to the report's fingerprint and to a named destination, and a script that refuses self-approval.

Everything kept forever. Symptom: thousands of run folders, including stopped test runs, with no deletion date. Fix: a retention line per kind of file, a person who deletes, and personal data kept out of the extracts from the start.

An account that expires. Symptom: the job stops on a Monday because the service account's password ran out. Fix: the expiry date in IT's calendar with a named person responsible, and the heartbeat to catch it if the reminder fails.

The runbook nobody has used. Symptom: the deputy opens it for the first time during an incident. Fix: a trace drill and a missed-run drill, done by the deputy, once a quarter, with the runbook fixed each time.

Do it yourself (4 hours, over three weeks)

1. Structure it (15 min). Write which numbers from your pipeline leave your team, where they go, how long they must be provable and who approves them.
2. Understand it (45 min). Try to trace one published number from last year. Write down every place the trail goes cold. Look at 24 months of your data for a change in meaning that passed every check. Rewrite your problem sentence.
3. Choose tools (15 min). Decide where run folders live and how they're backed up. Ask IT for a push monitor. Agree the drift rules and who approves.
4. Build it: run folders and heartbeat (60 min). Make every run write its folder with fingerprints, mark it read-only, and call the heartbeat at the end. Test the alert by pausing the job.
5. Build it: drift and approval (60 min). Write the drift check from approved runs, and the approval script that refuses self-approval and changed reports. Test both with a planted shift in a copy of the data.
6. Understand it: trace drill (30 min). Trace one current published number back to its rows on one page. Have your deputy trace a second one from the runbook alone.
7. Build it: retention and runbook (15 min). Write one retention line per kind of file with the person who deletes. Put the next quarterly drill in a shared calendar.

Exit check

Written finding: a trace of one number that was published from the pipeline, with:

• where it appears, and the approval that names that place;
• the run ID, and the fingerprints of the output, the source extract and the code version;
• the check and drift results for that run, with any explanation;
• the source rows that produced the number, and the field list that defined the view;
• the retention rule for each piece, and who deletes it.

It passes when every step names its run ID, fingerprint or person, the missed-run alert and the drift check have each fired once in a test, and a colleague repeats the trace from the runbook alone and reaches the same rows.

Rule to remember

Keep what each run read, and you can prove any number it published.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T10-L04 · Connect Your Analysis to a Live System, and Stop Bad Data Before It Reaches Anyone: the view, checks and run sheet this book keeps running.
• T12-L05 · Keep One List of Every AI System Your Team Runs, and Make It Survive You: the organisation-wide inventory, retention and handover.
• T15-L05 · Keep Your Group's AI Use Compliant and Running After You Leave: archive packages and a successor for research.
• T05-L05 · Keep Your Automations Running for Months, and After You Leave: the same monitoring for workflow automations.
• T04-L05 · Keep a Connected Knowledge Base Current, and Prove a Withdrawn Document Is Gone: tracing answers back to their sources in a document assistant.

Tools

• Uptime Kuma: Taught. The push monitor that notices a run that didn't happen.
• Git: Taught. Tags that name the code behind every run.
• KNIME: Catalogued. A visual alternative with its own run history.