Build a Caption Pipeline Your Team Can Rerun Every Week | Heidelberg AI Curriculum
T09-L03
Build a Caption Pipeline Your Team Can Rerun Every Week
Turn weekly recordings into reviewed captions without one person clicking through them: a small script that drafts captions with a pinned model and your glossary, a test set with known transcripts, a named reviewer for every item, and a README a deputy can follow. It passes when the deputy runs a week without you.
T09-L03 · Create media · Level 3 Builder · 17-minute read, plus 3 hours of doing
Forty talks, one person, one button
The institute records its Thursday seminar and puts the talks on the internal video page for people who missed them. Since spring the videos have captions, because a colleague with a hearing impairment asked, and because half the audience watches them on the train without sound. Sofia, the scientific coordinator, makes them. She uploads each recording to a transcription website, downloads the captions, fixes what she can, and uploads them next to the video. Then she's ill for two weeks. Nobody else knows the website, the login or which fixes she makes. When she's back, two talks have captions nobody checked, and one of them turns a speaker's "Lgr5-positive cells" into "large five positive cells" for forty minutes.
In a company it's the thirty how-to videos a software firm publishes for customers, in German. Every product release renames a few buttons, so the videos get re-recorded, and so do the captions. The support team makes them by hand, one video at a time, and after the spring release half the captions still use the old button names.
The transcription was never the hard part. The hard part is doing it the same way every week, checking it, and letting someone else take over. You are at Level 3 Builder: colleagues and customers depend on these captions, and a wrong one reaches all of them. This book shows you how to build a small caption pipeline, a set of steps that runs the same way every week, with a test set, named reviewers and a README, so the captions keep coming when you're away.
After this you can
Set up a manifest and a glossary so every recording has an ID, a language, a reviewer and the terms it must contain.
Draft captions with a pinned speech-to-text model through a small script that never overwrites a draft or an approved file.
Keep a test set of recordings with known transcripts, and a check that measures the error rate before any change goes live.
Route every draft to a named reviewer, and publish only what a person approved.
Hand the pipeline to a deputy with a README, and say where it stops and IT or self-hosting takes over.
Before you start
Books:T09-L01 · Make One Image or Narrated Clip, and Label It for AI labels and a first narrated clip. for pinning a environment with uv, which this pipeline reuses. for writing a script with OpenCode Desktop. The four-quadrant method is in .
Recordings you may process. Speakers must have agreed to recording and captioning. Before any recording goes to an outside service, check with your data protection contact which service your organisation has approved (T12-L01 · Decide What You May Paste Into AI at Work).
An OpenAI API key, the password a script uses to reach OpenAI's service, in a project your organisation controls, with a monthly spend limit. If the recordings must not leave your organisation, see Choose tools for the alternative.
Subtitle Edit or another caption editor, for reviewers.
Two colleagues: one who records three short test clips, and one who agrees to be the deputy.
About 3¼ hours: 17 minutes to read, about 3 hours to build and hand over, in two sessions.
Last verified 23 September 2026 (Heidelberg AI Desk line: 12 October 2026). OpenAI's speech-to-text service takes files up to 25 MB and costs well under a cent a minute. Data sent to it isn't used for training unless you opt in. Subtitle Edit is free and works offline. Details for experts at the end of this book.
The idea in one page
A caption pipeline has five parts. Leave one out and it depends on one person again.
Part
What it does
1. A manifest
One row per recording: ID, audio file, language, the terms it must contain, and who reviews it.
2. A glossary
The names, abbreviations and technical terms the model gets wrong, with a version number.
3. A drafting script
Sends each new recording to a pinned model (one fixed model name that changes only when you decide) with the glossary, and writes a caption draft. It never overwrites a draft or an approved file.
4. A test set with known answers
Three short recordings whose correct transcript you already have, and a check that measures how far the model's text is from it.
5. A review step and a README
A named person approves every caption file before it's published; the README lets a deputy run everything.
Four facts shape how you build it.
Speech-to-text fails in predictable places. Names, gene and product names, abbreviations, numbers and "not". A caption can be 97% right and still wrong in the one word the talk is about. The glossary helps the model; the review catches the rest.
Models change, and so does your setup. A provider can retire a model or change it under the same name; you change the glossary or the prompt. Any of these can make captions better or worse. The test set tells you which, in five minutes, before a week of talks goes out.
Error rate is a number you can compare. The word error rate counts the words a person would have to change, add or delete, divided by the words in the correct transcript. It's crude, but it moves when something breaks, and it's the same number for everyone who runs the check.
The model drafts; a person approves. No caption goes on the video page because the script finished. It goes there because a named reviewer read it against the video.
What stays yours: the glossary, the known transcripts, the error rate you accept and the decision to publish. The model writes drafts.
Worked example: two caption pipelines, four quadrants
Structure it. Sofia writes down what the institute expects. "Every Thursday one talk of about 50 minutes, in English, sometimes German. Captions on the internal video page within three working days. A good result: the captions are checked by someone who knows the field, and anyone in the coordination office can run the pipeline when I'm away."
Understand it. Before building anything, she runs one past talk through the obvious route: the whole recording sent to the OpenAI API (the service scripts talk to), captions back. Three things come out. The file is too big: an hour of the recorded audio is about 55 MB, over the 25 MB limit. The jargon fails as expected: "Lgr5" becomes "large five", a speaker's name is spelled three ways, and "scRNA-seq" never appears correctly. And reviewing the captions against the video takes her most of an afternoon. She can't do that every week, and she's the wrong person for it: she doesn't know which gene names are right.
The last point changes her problem more than the other two. Back to Structure it: "The pipeline drafts; a reviewer from the speaker's host group checks, because they know the terms. Their name goes into the manifest when the talk is booked. The host group also sends me ten to twenty terms from the talk a week ahead." The size problem has a simple fix: the AV team exports the talk as mono audio at 32 kbit/s, a smaller file that is still clear for speech, about 14 MB an hour. They also cut it at the end of the talk, because the audience in the Q&A never agreed to be captioned.
Choose tools. She picks the OpenAI API with whisper-1, because it returns captions with timings directly, as a WebVTT file (a standard caption format, .vtt), takes a prompt with terms, handles English and German, and costs well under a euro for a talk. gpt-transcribe gives cleaner text, but no caption timings, so it's the wrong model for this job. The institute's data protection contact approves the API for talks whose speakers agreed to captioning; unpublished data shown in a talk stays out, which the booking form now asks. The alternative, if recordings must never leave the institute, is the open-source Whisper model on your own hardware; T09-L04 · Transcribe Recordings That Must Stay in the Building shows how to run it. She writes the script with OpenCode Desktop, pins the environment with uv, the tool from T10-L03 · Build an Analysis Your Colleague Can Rerun and Get the Same Numbers that fixes which Python packages and versions the script uses, and gives reviewers Subtitle Edit, which opens a caption file next to the video.
Build it. She builds the test set first. Her colleague Malik records three clips of about two minutes each on the seminar-room microphone: one script packed with the institute's terms ("Lgr5-positive stem cells, scRNA-seq, organoids from the Kessler lab"), one with numbers and negations ("we did not see a difference at 37 degrees"), and one read with a strong accent by a colleague who volunteered. Because they read from scripts, the correct transcript is exact: it's the script. These three recordings and their scripts are the known answers.
Then the folder, the glossary and the settings:
seminar-captions/
README.md CHANGES.md settings.toml glossary.txt manifest.csv
transcribe.py check.py pyproject.toml uv.lock
tests/ three test clips (.mp3) and their scripts (.txt)
incoming/ this week's audio, exported by the AV team
drafts/ caption drafts and review-list.csv, written by the script
approved/ written only by reviewers
# settings.toml. Change here, then run check.py before the next real talk.
model = "whisper-1" # the only OpenAI model that returns captions with timings
glossary = "glossary.txt"
glossary_version = "2026-09"
prompt = "Scientific seminar at a biomedical research institute. Terms:"
test_language = "en"
max_wer = 0.08 # set from the first check run; see README
The glossary is a plain list, one term per line, kept short because whisper-1 accepts a prompt of at most 224 tokens (a token is a word or part of a word). Each talk's own terms go into the manifest instead, as expected_terms. The script adds them to that talk's prompt and checks the draft for them:
She asks OpenCode in Plan mode, where it proposes before it writes anything, for a script that reads these files, skips anything already drafted or approved, stops on files over 25 MB, and lists the expected terms missing from each draft. After review, transcribe.py does five things:
It reads the settings file, the glossary and the manifest.
For each recording in the manifest it skips the talk if a draft or an approved file with that ID already exists, so nothing is ever overwritten. It stops with a STOP message if the audio file is over 25 MB.
It sends the audio to the pinned model with the talk's language and a prompt made of the glossary plus the talk's own terms, and saves the captions it gets back as a draft.
It checks which of the talk's expected terms are missing from the draft and adds a line to the review list: ID, reviewer, fingerprint, model, glossary version and missing terms.
It prints one line per talk, saying who reviews it and which terms to listen for.
Both scripts find your API key in the terminal, where you set it once each time you open one, before you run them. Windows PowerShell comes first, then macOS and Linux:
$env:OPENAI_API_KEY="<your API key>"
export OPENAI_API_KEY="<your API key>"
The script itself:
"""Draft captions for every new recording in manifest.csv.
Run: uv run transcribe.py Settings: settings.toml Needs OPENAI_API_KEY."""
import csv, hashlib, pathlib, sys, tomllib
from openai import OpenAI
HERE = pathlib.Path(__file__).parent
S = tomllib.loads((HERE / "settings.toml").read_text(encoding="utf-8"))
lines = (HERE / S["glossary"]).read_text(encoding="utf-8").splitlines()
terms = [t.strip() for t in lines if t.strip() and not t.startswith("#")]
client = OpenAI()
with open(HERE / "manifest.csv", newline="", encoding="utf-8") as fh:
items = list(csv.DictReader(fh))
for item in items:
audio = HERE / "incoming" / item["audio_file"]
draft = HERE / "drafts" / f"{item['item_id']}.vtt"
if draft.exists() or (HERE / "approved" / draft.name).exists():
continue # never overwrite a draft or an approved file
if audio.stat().st_size > 25_000_000:
sys.exit(f"STOP: {audio.name} is over 25 MB. Ask the AV team for mono 32 kbit/s.")
fingerprint = hashlib.blake2b(audio.read_bytes(), digest_size=16).hexdigest()
talk_terms = [t for t in item["expected_terms"].split("|") if t]
with audio.open("rb") as f:
vtt = client.audio.transcriptions.create(
model=S["model"], file=f, response_format="vtt", language=item["language"],
prompt=S["prompt"] + " " + ", ".join(terms + talk_terms))
draft.parent.mkdir(exist_ok=True)
draft.write_text(vtt, encoding="utf-8")
missing = [t for t in talk_terms if t not in vtt]
with open(HERE / "drafts" / "review-list.csv", "a", newline="", encoding="utf-8") as fh:
csv.writer(fh).writerow([item["item_id"], item["reviewer"], fingerprint,
S["model"], S["glossary_version"], "; ".join(missing)])
print(f"{item['item_id']}: draft for {item['reviewer']}; missing terms: {missing or 'none'}")
The check runs the three test clips and compares the model's text with the scripts. It sends each clip to the same model with the same glossary. It counts the word error rate against the clip's script and lists glossary terms that are in the script but missing from the result. A clip passes when its rate is at or below the limit in the settings file and no term is missing, and the check prints PASS only when all three pass:
"""Transcribe the test clips and compare with their scripts. Run: uv run check.py"""
import pathlib, re, sys, tomllib
from openai import OpenAI
HERE = pathlib.Path(__file__).parent
S = tomllib.loads((HERE / "settings.toml").read_text(encoding="utf-8"))
lines = (HERE / S["glossary"]).read_text(encoding="utf-8").splitlines()
terms = [t.strip() for t in lines if t.strip() and not t.startswith("#")]
client = OpenAI()
def words(text):
return re.findall(r"[\w'-]+", text.lower())
def wer(ref, hyp):
"""Word error rate: words to change, add or delete, divided by words in ref."""
d = list(range(len(hyp) + 1))
for i, r in enumerate(ref, 1):
prev, d[0] = d[0], i
for j, h in enumerate(hyp, 1):
prev, d[j] = d[j], min(d[j] + 1, d[j - 1] + 1, prev + (r != h))
return d[-1] / len(ref)
ok = True
for script in sorted((HERE / "tests").glob("*.txt")):
ref = script.read_text(encoding="utf-8")
with script.with_suffix(".mp3").open("rb") as f:
got = client.audio.transcriptions.create(
model=S["model"], file=f, response_format="text",
language=S["test_language"], prompt=S["prompt"] + " " + ", ".join(terms))
rate = wer(words(ref), words(got))
lost = [t for t in terms if t.lower() in ref.lower() and t.lower() not in got.lower()]
passed = rate <= S["max_wer"] and not lost
ok = ok and passed
print(f"{script.stem}: error rate {rate:.1%}, missing terms {lost or 'none'} "
f"-> {'OK' if passed else 'TOO HIGH'}")
print("PASS" if ok else "FAIL: don't run real talks until this passes (README).")
sys.exit(0 if ok else 1)
Her first check fails on the term clip: the error rate is fine, but "scRNA-seq" is missing. The model writes "single-cell RNA seek". She adds the spoken form next to the term in the glossary, "scRNA-seq (single-cell RNA sequencing)", and removes a dozen terms nobody has used this year, to keep the prompt within its limit. The second run passes, with error rates between 3% and 6%. She sets max_wer = 0.08, just above the worst clip, and writes the reason into the README. From now on, any change to the model, the prompt, the glossary or the packages means running the check first, and a rate above 8% or a lost term means no real talks until someone has looked.
Then the review step. The script writes a line into review-list.csv for each draft: ID, reviewer, a fingerprint (a short code that identifies the exact audio file), model, glossary version, and the expected terms it couldn't find. Sofia sends the reviewer the draft and the video link. The reviewer opens both in Subtitle Edit, reads the captions against the talk, fixes them, saves the file into approved/ and adds their initials to the review list. Only files in that folder go on the video page, with the line "Captions drafted with AI and checked by [reviewer]."
The first real week shows the manifest paying off. The script reports missing terms: Kessler; the speaker said "the Keller lab" once and "Kessler" never, and the reviewer, from the Kessler lab, confirms the draft was right and the manifest was wrong. A missing term is a reason to listen, not a reason to change the caption.
Hand it over. The README covers setup (uv, the API key from the institute's project, never in a file), the weekly routine (manifest row, audio in incoming/, uv run transcribe.py, send drafts), the check (when and how, and what the 8% means), and When it stops, one line per STOP message. Maintainer: Sofia. Deputy: Malik, from the coordination office. The API project has a monthly spend limit of 20 euros, and the README says who can raise it.
Then the rerun. Sofia takes a week off on purpose. Malik follows the README on his own laptop: installs uv, runs uv run check.py, gets PASS, and processes that week's talk. One thing stops him: the AV team sent the full recording, 52 MB, and the script refuses it. He finds the STOP line in the README and asks for the talk-only export. The talk is captioned on Friday, reviewed by Monday, and Sofia reads about it in review-list.csv when she's back.
What it taught her: the script took an evening. Deciding who reviews, and writing the test set that tells her when something changed, took the rest, and they're why the pipeline survived her holiday.
Company: German captions for how-to videos
Structure it. Jana in customer support writes: "Thirty how-to videos in German, two to five minutes each, re-recorded after every product release. Captions within a week of the release, in the product's current words."
Understand it. She compares last release's captions with the current product. The model does fine with ordinary German. It fails on the product's own words: "Arbeitsbereich", "Freigabe-Workflow" and the product name, which it spells like an English word. Worse, the spring release renamed "Projekt" to "Arbeitsbereich", and several captions were corrected by hand to the old term, because that's what the reviewer remembered. Back to Structure it: the glossary is versioned per release and comes from the release notes, not from memory. Each video's ID includes the release, HOWTO-12-R26.3, so an approved caption can never be matched to the wrong recording.
Choose tools. The same script and model, with language = "de". The company is an OpenAI API customer and already has a project with European data residency, meaning its data is kept in Europe, which the data protection contact prefers; the README records the European address the script uses. Her test set is three videos from the last release whose captions were approved, and she re-records the narration of one with the new terms, so its script is the known answer for the new glossary.
Build it. Her first check fails: "Freigabe-Workflow" comes back as "Freigabe Workflow" in two places. She decides that's fine, because a reader doesn't stumble over a missing hyphen, and changes her term check to ignore hyphens rather than raising max_wer. That decision goes into CHANGES.md with the reason. The reviewer for each video is the product specialist for that area, named in the manifest. Her deputy runs the next release from the README.
Where it stops. After the second release, the web team asks whether approved captions could upload themselves to the help-centre video platform. That's a connection into a system the company runs, with a service account (a login for a program, not a person) and permissions: Level 4 work with IT. Jana writes it into the README as the next step.
In both cases the model was the easy part. The test set, the glossary per version and a named reviewer are what made the captions dependable.
Where your build stops
A script, a test set, a manifest and a README are right for a team that captions a few hours a week, by hand, with named reviewers. Stop and bring in IT, or move to Level 4, when:
recordings may not leave your organisation, such as patient consultations, participant interviews or confidential meetings. Then the model has to run on your own hardware (T09-L04 · Transcribe Recordings That Must Stay in the Building), and an ethics or data protection review comes first;
captions should publish themselves to a video platform, learning system or website;
the volume grows to many hours a week, where the manual review needs its own planning and budget;
Captions published without review.Symptom: a gene name is wrong for forty minutes on the video page. Fix: only files in approved/ are published, and only reviewers write there.
The wrong reviewer.Symptom: the coordinator checks captions she can't judge. Fix: a reviewer who knows the field, named in the manifest when the recording is booked.
Changing the setup without the check.Symptom: a new glossary makes one term right and three others wrong. Fix: any change to model, prompt, glossary or packages means check.py first. A FAIL stops real work until someone has looked.
Known answers you guessed.Symptom: the check passes, but the reference transcript had errors of its own. Fix: use scripted test clips, where the script is the correct transcript, or a transcript a second person corrected against the recording.
A glossary that grows forever.Symptom: two hundred terms, and the prompt runs past the model's limit. Fix: keep the standing glossary short and put each recording's terms into the manifest.
Forcing the glossary onto the caption.Symptom: a script replaces every "Keller" with "Kessler". Fix: a missing term raises a flag for the reviewer. It never changes the text automatically.
Recordings nobody cleared.Symptom: the Q&A, with audience voices, goes to an outside service. Fix: process only what speakers agreed to, and cut the rest before the audio reaches the script.
Do it yourself (3 hours, two sessions)
Structure it (15 min). Write who needs captions, how fast, in which languages, and who can judge their accuracy.
Understand it (30 min). Run one past recording through the API by hand. List what breaks: file size, terms, languages, review time. Decide who reviews, and rewrite your problem.
Choose tools (15 min). Confirm with your data protection contact which service is approved for your recordings. Pick the model for captions with timings, and decide where drafts and approved files live.
Build it: test set and check (40 min). Record three scripted clips with your hardest terms, numbers and a negation. Set up the folder with uv, write the glossary and run check.py. Set max_wer just above your worst clip and write down why.
Build it: script and review (45 min). Draft transcribe.py with OpenCode, review every line, and run one real recording. Send it to the named reviewer, and publish only what comes back approved.
Build it: README and rerun (35 min). Write the README and set a spend limit. Let your deputy run the check and one week from the README alone. Fix the README wherever they got stuck.
Exit check
What you hand in: a pipeline folder and one week of captions, with:
the settings file with the pinned model, the glossary with a version, and the manifest with a reviewer for every item;
the drafting script, which never overwrites a draft or an approved file;
three test clips with known transcripts, and the check script with the error-rate limit and its reason;
the review list and the approved caption files for one week;
a README with setup, weekly routine, check and STOP messages, a maintainer, a deputy and the spend limit.
It passes when the deputy runs the check and one week's recordings on their own computer from the README alone, the check passes, and every published caption file has a named reviewer.
Rule to remember
The model drafts, the test set tells you when it changed, and a named person approves.
Details for experts
Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.
What
Details
What it means for you
Models
whisper-1 returns captions with timings, in the formats json, text, srt, verbose_json and vtt. gpt-transcribe is recommended for plain text (json, text, diarized_json, verbose_json). gpt-4o-transcribe-diarize adds speaker labels, known as diarization. Timestamp granularities (timings per word or segment) work only with whisper-1.
Keep whisper-1 pinned in the settings file for captions.
File limit
Up to 25 MB per file, as mp3, mp4, mpeg, mpga, m4a, wav or webm.
Ask for mono audio at 32 kbit/s, so an hour fits.
Prompt limit
The prompt for whisper-1 holds at most 224 tokens.
Keep the standing glossary short; each talk's terms go in the manifest.
Prices
Whisper: $0.006 a minute. gpt-transcribe: $0.0045 a minute.
A 50-minute talk costs about 30 cents with Whisper.
Data use
Data sent to the API isn't used for training unless you opt in. Abuse-monitoring logs, kept to detect misuse, are stored for up to 30 days.
Mention the 30 days to your data protection contact.
European data residency
Audio transcription is eligible, with data kept in the EEA (the EU plus Iceland, Liechtenstein and Norway) and Switzerland. It needs approval from OpenAI's sales team, a Modified Retention amendment (a change to the contract) and a project created with the Europe region, which uses eu.api.openai.com.
This is Jana's route; write the address into the README.
Subtitle Edit
Free, open source under the MIT licence, works offline, runs on Windows, macOS and Linux.
OpenCode: Taught. Writes the first draft of the script.
Whisper (open source): Catalogued. The open model behind whisper-1, for running on your own hardware.
Heidelberg AI Desk: Catalogued. Runs reviewed AI apps on your own computer.
Deepgram: Catalogued. Another speech-to-text service with term prompting.
n8n: Catalogued. When you'd rather build the same steps as a visual workflow.
Watch
13 minHow to Install & Use Whisper AI Voice to Text
How to Install & Use Whisper AI Voice to TextKevin Stratvert · 13 min
T09-L03: See the local Whisper alternative transcribe files and produce text and subtitle outputs, then check the result against the recording. This is not the book's hosted API pipeline; installation and model-default advice reflect the recording.
T09-L04: See local speech recognition with an explicitly selected model and source language. The video uses the original Whisper CLI, while this book uses faster-whisper. Follow the book for current setup, network isolation, permissions and deletion.
