Heidelberg AI Curriculum

T04-L03 · Your documents & knowledge · Level 3 Builder · 17-minute read, plus 3 hours of doing

Right most days, wrong on the day it mattered

Six months ago a postdoc in a proteomics group put the group's 31 standard operating procedures into a document assistant on the lab PC. It works well. People ask it before they prepare samples, and it answers with a citation. Then a master's student asks how long the standard digest runs. The answer says overnight, with a citation to the right SOP. The SOP says four hours, and that the digest may be extended overnight for membrane fractions. Twelve samples are prepared the wrong way before anyone notices.

In a company it's a customer service team with a shared assistant over its policies and supplier handbook. It answered well for months. Then the handbook grew to sixty pages, and answers about delivery times started quoting the wrong table. Nobody could say when it began.

You are at Level 3 Builder: colleagues act on what your assistant says, so a wrong answer becomes their mistake. At Level 2 you made sure the right documents were in the set (T04-L02). This book is about what happens after that: the right document is in the set, and the answer is still wrong. You'll write questions you know the answers to, find out which part failed, fix one thing at a time, and leave a check a colleague can run without you.

After this you can

Write a test set of ten to fifteen questions with the passage and answer you expect, including one the documents can't answer.
Tell from the retrieved passages whether a wrong answer came from the search, from the answer or from the question.
Change one document or one retrieval setting at a time and show which test questions it fixed and which it broke.
Leave a settings record, a README and a check that a colleague reruns after every document change.
Say where your build stops and IT's work begins.

Before you start

Books: T04-L02 for the rule and register that decide what goes into the set. T03-L03 for testing with fixed cases. T12-L01 for what may go into which tool.
A document assistant your group already uses, with a curated set from T04-L02. The lab example uses AnythingLLM Desktop with a local model. The company example uses a shared Claude project.
Python 3.10 or newer if you want the scripted check in the lab example. The company example needs only a spreadsheet.
The person who writes the documents, to confirm the expected answers, and one colleague who will rerun the check.
About 3 hours 20 minutes: 17 to read, about 3 hours to build and hand over, spread over two sessions.

Last verified 23 September 2026. AnythingLLM Desktop is a free download for Mac, Windows and Linux, for one user. Claude projects are shared on Team and Enterprise plans and, on paid plans, search their files automatically once they grow large. Details for experts at the end of this book.

The idea in one page

A document assistant does two jobs, one after the other. First it searches, a step also called retrieval: it cuts every document into passages (chunks), and for each question it picks the few passages that match best. Then it answers: a language model writes a reply from those passages only. A wrong answer can come from either job, or from a third place: the question itself.

What went wrong	What you see	Where to fix it
The search missed	The right passage isn't among the passages the model received.	The document (headings, tables, one topic per section), then retrieval settings.
The answer went wrong	The right passage arrived, and the answer says something it doesn't: may became must, a condition vanished.	The instruction to the model, then the model.
The question has no single answer	Two current documents answer it differently, or the question mixes two things.	Your test, and what the assistant should do with such questions.

You can only tell these apart if you look at the passages. A citation badge shows which file was used. It doesn't show whether the sentence you need was in the passage the model read.

Four habits make the difference.

Write the test before you change anything. Ten to fifteen questions people actually ask, each with the document, the sentence that answers it and the short phrase a correct answer must contain. Include a table value, a may or except, an exact identifier, and one question the documents can't answer. The person who writes the documents confirms the expected answers.
Fix the document before the setting, and the setting before the model. Most search misses come from how a document is written: a table row without its column headers, a rule buried three pages below its heading, a scanned PDF with no text in it. Fixing the document helps every tool you'll ever use. A new model rarely fixes a search miss, because the model never saw the passage.
Change one thing, rerun everything. After each change, run all the questions again and compare question by question. A change that fixes one question and breaks another is common. A total that goes up can hide it.
Write it down so someone else can rerun it. A settings record, the test file, a README, and a named person who runs the check after every document change. The check is what tells your colleagues the assistant still works.

What stays yours: which questions matter, what the right answer is, and whether a failure is serious enough to stop people using the assistant. The tool finds and writes. You decide what correct means.

Worked example: two teams, four quadrants

Both teams use the four-quadrant method from T01-L01. The lab builds a scripted check; the company does the same work by hand in a spreadsheet.

Lab: the SOP assistant on the prep bench PC

Structure it. Jonas, the postdoc who set up the assistant, writes: "Colleagues ask the SOP assistant before they prepare samples. A useful result: for the questions people really ask, the right passage reaches the model and the answer says only what the SOP says. When the SOPs don't say, it says so. And Mira, our technician, can check all of that in fifteen minutes without me."

Understand it. He starts with the questions, not the settings. He reads the last two months of the workspace's chat history and the lab's chat channel, and picks twelve questions people actually asked. For each one he writes the SOP, the sentence that answers it and a short phrase a correct answer must contain. The group leader, who signs the SOPs, confirms each expected answer. One question has no answer in the SOPs on purpose: "Who approves buying a new column?" That's decided by the group leader, and no SOP says so.

Then he asks all twelve by hand, and for each answer opens the citations and clicks the file, so he can read the passages the model received, each with its match score. Eight are right. Four aren't, and they aren't wrong in the same way:

Q03, the enzyme-to-protein ratio: the passage that arrived was a table row reading Trypsin | 1:50 | 37 | 4. Without the column headers the model guessed which number was which. The right SOP, the wrong usable text: a search problem caused by the document.
Q05, digest time: the right passage arrived. It says four hours, and that the digest may be extended overnight for membrane fractions. The answer said overnight. An answer problem.
Q08, who signs off a new LC column: the name sits in SOP-02, three paragraphs below a heading about column storage. The passage with the name never reached the model. A search problem.
Q10, "How long do we digest?": he expected four hours, but two current SOPs answer it: the standard digest and the in-gel digest, with different times. The assistant picked one. Neither the search nor the answer was wrong. The question was.

Q10 sends him back to Structure it. His first statement said "the right passage", as if every question had one. It doesn't. He rewrites it: "When a question matches more than one current SOP, the answer names each SOP and its value, and asks which one you mean." He rewrites the expected answer for Q10 so it must mention both SOPs, and adds one question for the in-gel digest. That makes thirteen.

Choose tools. He keeps AnythingLLM Desktop. The SOPs contain unpublished methods the institute hasn't approved for cloud tools, and the desktop app runs a local model on the lab PC. It also shows the passages behind every answer, has the retrieval settings he needs per workspace, and includes an API, a way for a script to send it questions, so the check can be a script. If colleagues later need the assistant from their own desks, Open WebUI through Heidelberg AI Desk has the same kinds of settings and is the alternative to try.

What he doesn't add: a reranker (a second pass that re-sorts the passages), a second model, or an automatic job that reprocesses documents. None of the four failures asks for them. The check itself is a short Python script and a CSV file, a plain table any spreadsheet opens, because Mira should be able to read both.

Build it. He saves the test as tests.csv, one row per question:

id,question,expect_doc,expect_passage,expect_answer,forbid
Q03,What enzyme-to-protein ratio do we use for the standard digest?,SOP-12,1:50,1:50,
Q05,How long does the standard digest run?,SOP-12,four hours,4 h,
Q08,Who signs off a new LC column?,SOP-02,signed off by,Mira,
Q12,Who approves buying a new column?,-,-,Not in the current SOPs,

expect_passage is a phrase that must appear in a passage from the expected SOP. expect_answer is a phrase every correct answer contains. forbid catches answers that must never appear; he fills it for the digest question once he has seen how it went wrong. For Q12, the dash means no passage should answer it, and the answer must be the refusal text.

He sets the workspace to Query mode, so it answers only from the SOPs, sets the Query mode refusal response to Not in the current SOPs – ask Jonas or Mira. and turns LLM Temperature down, so answers vary less between runs. The workspace instruction now reads:

Answer only from the SOP passages you are given. Name the SOP and section.
Keep "may", "must", "except" and every number exactly as the SOP states them.
If two SOPs give different answers, list both with their SOP numbers and
ask which one the user means.
If the passages don't answer the question, reply exactly:
Not in the current SOPs – ask Jonas or Mira.

The check script sends each question through the API and scores the two jobs separately:

"""Ask every question in tests.csv and score search and answer separately.
Run:  python check.py      (AnythingLLM must be open; key in ALLM_KEY)"""
import csv, json, os, sys, urllib.request

API = "http://localhost:3001/api/v1/workspace/sop-assistant/chat"
KEY = os.environ["ALLM_KEY"]


def ask(question):
    body = json.dumps({"message": question, "mode": "query"}).encode()
    req = urllib.request.Request(API, data=body, headers={
        "Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=600) as r:
        return json.load(r)


def norm(text):
    return " ".join((text or "").lower().split())


failures = 0
with open("tests.csv", newline="", encoding="utf-8") as fh:
    for t in csv.DictReader(fh):
        reply = ask(t["question"])
        answer = norm(reply.get("textResponse"))
        found = [(s.get("title", ""), norm(s.get("text") or s.get("chunk")))
                 for s in reply.get("sources", [])]
        if t["expect_doc"] == "-":
            search = "n/a"
        elif any(t["expect_doc"] in title and norm(t["expect_passage"]) in text
                 for title, text in found):
            search = "OK"
        else:
            search = "MISS"
        good = norm(t["expect_answer"]) in answer
        if t["forbid"] and norm(t["forbid"]) in answer:
            good = False
        failures += search == "MISS" or not good
        print(f"{t['id']}: search {search:4}  answer {'OK' if good else 'WRONG'}")
        if search == "MISS" or not good:
            print("   answer:", answer[:200])
            for title, text in found:
                print("   passage from", title, ":", text[:120])
print("PASS" if failures == 0 else f"FAIL: {failures} question(s), see README")
sys.exit(1 if failures else 0)

He had OpenCode write the first draft from a description of the API response, then checked the field names against the API documentation page his own installation serves at /api/docs. The script prints the passages for every failure, because that's where the diagnosis happens.

Now one change at a time, rerunning all thirteen questions after each.

Change 1, the document for Q03. He rewrites the ratio table in SOP-12 so each row carries its headers: Standard digest: trypsin, enzyme-to-protein ratio 1:50, 37 °C, 4 h. He removes the old file from the workspace and adds the new one, because a changed document is only searched once it has been embedded again: turned into the numbers the search compares by meaning. Q03 now passes. Nothing else moves.

Change 2, the instruction for Q05. The may line in the instruction above goes in, and he adds overnight to the forbid column for Q05. Q05 passes.

Change 3, a setting for Q08. He raises Max Context Snippets from 4 to 8, reasoning that more passages will include the one with the name. Q08 passes. Q12 fails: with eight passages, one loosely related paragraph about column orders reached the model, and it wrote a confident answer naming the lab manager. That's the result the per-question comparison exists for. The total stayed at 12 of 13, which hides one fix and one new fault. He sets the value back to 4.

Change 4, the document for Q08. He moves the sign-off rule in SOP-02 into its own short section with the heading Signing off a new LC column, and embeds the SOP again. Q08 passes, Q12 still refuses. All thirteen pass.

He writes the settings record, settings.md, next to the test: the AnythingLLM version, the model names, every retrieval and chat setting the workspace shows (the list is in Details for experts), the instruction and the refusal text, each with the date he last changed it. Then the README, the page a colleague follows to run the check:

# SOP assistant: how to check it

Maintainer: Jonas. Deputy: Mira. Expected answers confirmed by the group
leader on <date>. Settings: settings.md. Never change a setting without
rerunning the check.

## Run the check (about 15 minutes on the lab PC)
1. Open AnythingLLM. Leave it open.
2. Open a terminal in this folder and run:
   set ALLM_KEY=<key from the lab password manager>   (macOS/Linux: export)
   python check.py
3. PASS: done. Write the date in CHANGES.md.

## Run it after every document change
Add or replace the SOP in the workspace, then run the check before you
tell anyone the new SOP is in.

## When the check fails
- search MISS: read the passages it printed. Usually the document needs a
  heading closer to the rule, or a table row needs its column names.
  Fix the document, embed it again, rerun.
- answer WRONG with the right passage: the model changed the meaning.
  Tell Jonas. Don't change the test to make it pass.
- The same question passes and fails on repeated runs: note it in
  CHANGES.md and tell Jonas.
- Until it passes, put a note on the PC: "SOP assistant: check SOP-xx by hand."

## Changing a test
Only with the group leader's OK, and write why in CHANGES.md.

The test that matters comes three weeks later. Mira adds a new SOP for phospho-enrichment and runs the check from the README, with Jonas out of the room. Q03 fails with a search miss: the new SOP also has a ratio table, and its row now comes first. She reads the printed passages, sees the pattern the README describes, gives the new table's rows their column names, embeds it again and gets PASS. Her one question for Jonas was where the API key lives; the README now says.

Where his build stops. The group's assistant runs on one PC, for one group, with SOPs the group writes. Two requests would take it past that. Colleagues want it on their own desks with personal logins, and the institute wants it to read SOPs directly from its document management system, where each group sees only its own. Both need accounts, access rights and a server someone keeps running: that's the institute's IT, and T04-L04. Jonas brings them his test set and README, which is the most valuable part of what he built.

What it taught him: the fixes took an hour. Writing thirteen questions with confirmed answers took two, and that's what made every later change safe.

Company: the policy assistant after the handbook grew

Structure it. Markus leads a customer service team of five who answer customers from a shared Claude project with their policies and the supplier handbook (T04-L02). "My team quotes the assistant's answers to customers. A useful result: for our twenty most common questions it gives the current rule with the right clause, and says when our documents don't answer. My deputy Selin can check that without me."

Understand it. Since the handbook grew to sixty pages, the project shows the indicator that it now searches its files instead of reading them all. That's when answers about delivery times went wrong. The project has no retrieval settings to change, and it doesn't show the passages it found. So he changes the instruction to make the search visible: "Quote the passage you used, word for word, with file name and section." The quote tells him what the search found.

He writes twelve test questions in a spreadsheet with the columns question, expected file and section, phrase the answer must contain, quoted passage, search OK?, answer OK?, and asks them one by one. Two failures are searches: the delivery table in the handbook reads DE | 2 | 4 | 5 in the quoted passage, with no headers, and the answer confuses standard and express delivery. One is a question: "How fast do we ship to Austria?" has two answers, standard and express. Back to Structure it, like Jonas: questions with two answers get both, labelled.

Choose tools. He stays with the Claude project, which is what his company has approved. His levers are the documents and the instruction. He splits the handbook into one file per chapter, named by chapter title, and rewrites the delivery table as one line per country and service: Austria, express: 2 working days. He asks the policy lead to confirm the new file matches the old table.

Build it. He reruns all twelve questions. Eleven pass; one answer quotes the right passage and then rounds "10 working days" to "two weeks". He adds "Keep every number and unit exactly as written" to the instruction and reruns: twelve of twelve. The spreadsheet gets a tab with the settings record: plan, files with dates, the instruction text. A short README at the top says who maintains it, when to run the sheet (after every document change, and on the first Monday of the month), and what to do on a failure: stop quoting that topic from the assistant, tell Markus, and answer from the policy file until it's fixed. Selin runs the sheet the next month on her own, in 25 minutes. The only thing she changed was a question the supplier had made obsolete, and she asked the policy lead first.

The stop line is the same as in the lab. When sales asks for the assistant inside Outlook, drafting replies from SharePoint with each person's own access rights, that's a connection to company systems. Markus sends the request to IT with his test sheet attached (T14-L03, T04-L04).

What goes wrong

Judging answers without looking at passages. Symptom: you try three prompt wordings for a wrong answer and none helps. Fix: open the passages first. If the right sentence never arrived, no instruction can use it. Fix the document or the search.

Tables that lose their headers. Symptom: the answer has the right numbers in the wrong places. Fix: rewrite table rows so each carries its column names, or keep the table short enough to stay in one passage. Add a table question to the test set.

Several changes at once. Symptom: the score goes up after new chunking, more snippets and a new model, and nobody knows which one helped, or what else it changed. Fix: one change, then all questions again, compared one by one.

A higher total that hides a new fault. Symptom: 12 of 13 before and after, and a question that used to refuse now invents. Fix: list which questions changed, in both directions, after every change.

A test that bends to pass. Symptom: expected answers quietly change after a failure. Fix: only the person who confirms the documents may change an expected answer, and the change is written down with its reason.

No question the documents can't answer. Symptom: the assistant answers everything, including questions nobody documented. Fix: keep at least one absent question in the set, and check it after every setting change. More passages make invented answers more likely.

A changed document that still gives old answers. Symptom: you edited the SOP, the answer stays the same. Fix: remove and re-add the document so it's embedded again, then rerun the check. A changed chunk size also applies only to documents embedded afterwards.

Do it yourself (3 hours, two sessions)

Structure it (15 min). Write two sentences: who acts on the assistant's answers, and what a correct answer must do, including what happens when the documents don't say.
Understand it (60 min). Collect ten to fifteen real questions. For each, write the document, the answering sentence and the phrase a correct answer must contain; include a table, a may or except, an identifier and one absent question. Get the expected answers confirmed. Ask them all, read the passages behind every answer, and label each failure search, answer or question. If a question has two answers, rewrite your sentences from step 1.
Choose tools (15 min). Decide what you can change: documents, instruction, which retrieval settings. Choose between a scripted check (if your tool has an API) and a spreadsheet run by hand.
Build it: one change at a time (45 min). Fix documents first. After each change, rerun every question and write down what moved in both directions. Keep a change only if nothing broke.
Build it: record and README (30 min). Write the settings record and a README: who maintains it, how to run the check, when to run it, and what to do on each kind of failure.
Build it: the rerun (15 min). Ask your colleague to change or add one document and run the check from the README while you stay out of the room. Fix the README wherever they got stuck.

Exit check

Delivered artifact: a test folder or sheet for your group's document assistant with:

ten to fifteen test questions, each with the expected document, passage phrase and answer phrase, confirmed by the person responsible for the documents, including one the documents can't answer;
the settings record: tool and version, model, retrieval settings you can see, instruction and refusal text, with dates;
a before-and-after result for one change, listing which questions it fixed and which it broke;
a README with the maintainer, a deputy, when to run the check and what to do on a search miss, a wrong answer and an ambiguous question;
your colleague's rerun after they changed or added a document.

It passes when your colleague's rerun ends in PASS, or every failure is labelled search, answer or question with its next step, and they needed nothing beyond the README.

Rule to remember

Read the passages before you fix anything, and change one thing at a time.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

What	Details	What it means for you
AnythingLLM Desktop	Current release v1.16.2. It has a built-in model provider (runs the language model) and embedder (turns passages into searchable numbers). It serves one user; logins for several users need the Docker version, a server edition.	One lab PC is fine; personal logins are IT's job (T04-L04).
API key	Settings → Developer API → Generate New API Key.	Keep it in the password manager; the script reads it from `ALLM_KEY`.
API address	`http://localhost:3001/api`, documentation at `/api/docs`. The chat call `POST /v1/workspace/{slug}/chat` (slug: the workspace's short name) with `"mode": "query"` answers only from the workspace's documents. The answer comes back in `textResponse`, the passages in `sources`.	These are the names the check script relies on. If an update renames them, compare with `/api/docs`.
Chat settings	Workspace gear icon → Chat Settings: chat mode, Query mode refusal response, LLM Temperature (how much answers vary between runs).	Where Jonas set Query mode and the refusal text.
Search settings	Gear icon → Vector Database, the store the search runs on. Max Context Snippets: passages sent to the model, recommended 4. Document similarity threshold: how close a passage must match (No restriction, Low ≥ .25, Medium ≥ .50, High ≥ .75). Search Preference: its Accuracy Optimized reranking works only with LanceDB.	Change one at a time and rerun, as in Change 3.
Chunking	Settings → AI Providers → Text Splitter & Chunking. Text Chunk Size, the length of each passage (default 1,000 characters); Text Chunk Overlap, how much neighbouring passages share (default 20). A change applies only to documents embedded afterwards.	After changing it, remove and re-add every document.
Claude projects	Shared on Team and Enterprise with Can view or Can edit. On paid plans (Pro, Max, Team, Enterprise) a project switches to searching its files near the context limit (the most text the model reads at once), shows an indicator, and has no settings for it.	Your levers are the documents and the instruction, as for Markus.
Open WebUI	Chunk size, Top K (passages sent, like Max Context Snippets) and hybrid search (meaning plus keywords) under Admin → Settings → Documents.	The same one-change-at-a-time tests work there.

Test and Fix the Document Assistant Your Group Relies On