Last verified 23 September 2026. AnythingLLM Desktop is a free download for Mac, Windows and Linux, for one user. Claude projects are shared on Team and Enterprise plans and, on paid plans, search their files automatically once they grow large. Details for experts at the end of this book.

The idea in one page

A document assistant does two jobs, one after the other. First it searches, a step also called retrieval: it cuts every document into passages (chunks), and for each question it picks the few passages that match best. Then it answers: a language model writes a reply from those passages only. A wrong answer can come from either job, or from a third place: the question itself.

You can only tell these apart if you look at the passages. A citation badge shows which file was used. It doesn't show whether the sentence you need was in the passage the model read.

Four habits make the difference.

1. Write the test before you change anything. Ten to fifteen questions people actually ask, each with the document, the sentence that answers it and the short phrase a correct answer must contain. Include a table value, a may or except, an exact identifier (a code or name that must match letter for letter, such as an SOP number), and one question the documents can't answer. The person who writes the documents confirms the expected answers.
2. Fix the document before the setting, and the setting before the model. Most search misses come from how a document is written: a table row without its column headers, a rule buried three pages below its heading, a scanned PDF with no text in it. Fixing the document helps every tool you'll ever use. A new model rarely fixes a search miss, because the model never saw the passage.
3. Change one thing, rerun everything. After each change, run all the questions again and compare question by question. A change that fixes one question and breaks another is common. A total that goes up can hide it.
4. Write it down so someone else can rerun it. A settings record, the test file, a README, and a named person who runs the check after every document change. The check is what tells your colleagues the assistant still works.

What stays yours: which questions matter, what the right answer is, and whether a failure is serious enough to stop people using the assistant. The tool finds and writes. You decide what correct means.

Worked example: two teams, four quadrants

Both teams use the four-quadrant method from T01-L01 · From Idea to Working Result: The Four-Quadrant Method. The lab builds a scripted check; the company does the same work by hand in a spreadsheet.

Lab: the SOP assistant on the prep bench PC

Structure it. Jonas, the postdoc who set up the assistant, writes: "Colleagues ask the SOP assistant before they prepare samples. A useful result: for the questions people really ask, the right passage reaches the model and the answer says only what the SOP says. When the SOPs don't say, it says so. And Mira, our technician, can check all of that in fifteen minutes without me."

Understand it. He starts with the questions, not the settings. He reads the last two months of the workspace's chat history (a workspace is AnythingLLM's name for one document collection with its own chats and settings) and the lab's chat channel, and picks twelve questions people actually asked. For each one he writes the SOP, the sentence that answers it and a short phrase a correct answer must contain. The group leader, who signs the SOPs, confirms each expected answer. One question has no answer in the SOPs on purpose: "Who approves buying a new column?" That's decided by the group leader, and no SOP says so.

Then he asks all twelve by hand, and for each answer opens the citations and clicks the file, so he can read the passages the model received, each with its match score. Eight are right. Four aren't, and they aren't wrong in the same way:

• Q03, the enzyme-to-protein ratio: the passage that arrived was a table row reading Trypsin | 1:50 | 37 | 4. Without the column headers the model guessed which number was which. The right SOP, the wrong usable text: a search problem caused by the document.
• Q05, digest time: the right passage arrived. It says four hours, and that the digest may be extended overnight for membrane fractions. The answer said overnight. An answer problem.
• Q08, who signs off a new LC column: the name sits in SOP-02, three paragraphs below a heading about column storage. The passage with the name never reached the model. A search problem.
• Q10, "How long do we digest?": he expected four hours, but two current SOPs answer it: the standard digest and the in-gel digest, with different times. The assistant picked one. Neither the search nor the answer was wrong. The question was.

Q10 sends him back to Structure it. His first statement said "the right passage", as if every question had one. It doesn't. He rewrites it: "When a question matches more than one current SOP, the answer names each SOP and its value, and asks which one you mean." He rewrites the expected answer for Q10 so it must mention both SOPs, and adds one question for the in-gel digest. That makes thirteen.

Choose tools. He keeps AnythingLLM Desktop. The SOPs contain unpublished methods the institute hasn't approved for cloud tools, and the desktop app runs a local model on the lab PC. It also shows the passages behind every answer, has the retrieval settings he needs per workspace, and includes an API, a way for a script to send it questions, so the check can be a script. If colleagues later need the assistant from their own desks, Open WebUI on a server (T11-L04 · Put Your Team's Assistant on a Server Colleagues Can Reach) has the same kinds of settings and is the alternative to try.

What he doesn't add: a reranker (a second pass that re-sorts the passages), a second model, or an automatic job that reprocesses documents. None of the four failures asks for them. The check itself is a short Python script and a CSV file, a plain table any spreadsheet opens, because Mira should be able to read both.

Build it. He saves the test as tests.csv, one row per question:

expect_passage is a phrase that must appear in a passage from the expected SOP. expect_answer is a phrase every correct answer contains. forbid catches answers that must never appear; he fills it for the digest question once he has seen how it went wrong. For Q12, the dash means no passage should answer it, and the answer must be the refusal text.

Under the workspace's gear icon, in Chat Settings, he sets the chat mode to Query, so it answers only from the SOPs, sets the Query mode refusal response to Not in the current SOPs – ask Jonas or Mira. and turns LLM Temperature down, so answers vary less between runs. The workspace instruction now reads:

For the script he creates an API key, a password that lets a script use AnythingLLM, under Settings → Developer API → Generate New API Key, and stores it in the lab password manager.

The check script sends each question through the API and scores the two jobs separately. You don't need to read Python to use it; this is what it does:

1. It reads tests.csv one row at a time and sends each question to AnythingLLM on this PC (localhost:3001 is the address of AnythingLLM on this computer), using the key it finds in ALLM_KEY.
2. From each reply it keeps two things: the answer text and the passages the model received. It lowercases both and removes extra spaces, so small differences in spelling don't count as failures.
3. It scores the search: OK if a passage from the expected SOP contains the expected phrase, MISS if not, and n/a for the question the SOPs can't answer.
4. It scores the answer: OK if the answer contains the expected phrase and not the forbidden one. For every failure it prints the answer and the passages, because that's where you find the cause.
5. At the end it prints PASS, or FAIL with the number of failed questions.

He had OpenCode write the first draft from a description of the API response, then checked the field names against the API documentation page that his own installation shows in the browser at http://localhost:3001/api/docs. The script prints the passages for every failure, because that's where the diagnosis happens.

Now one change at a time, rerunning all thirteen questions after each.

Change 1, the document for Q03. He rewrites the ratio table in SOP-12 so each row carries its headers: Standard digest: trypsin, enzyme-to-protein ratio 1:50, 37 °C, 4 h. He removes the old file from the workspace and adds the new one, because a changed document is only searched once it has been embedded again: turned into the numbers the search compares by meaning. Q03 now passes. Nothing else moves.

Change 2, the instruction for Q05. The may line in the instruction above goes in, and he adds overnight to the forbid column for Q05. Q05 passes.

Change 3, a setting for Q08. He raises Max Context Snippets, the number of passages the model receives, from 4 to 8, reasoning that more passages will include the one with the name. Q08 passes. Q12 fails: with eight passages, one loosely related paragraph about column orders reached the model, and it wrote a confident answer naming the lab manager. That's the result the per-question comparison exists for. The total stayed at 12 of 13, which hides one fix and one new fault. He sets the value back to 4.

Change 4, the document for Q08. He moves the sign-off rule in SOP-02 into its own short section with the heading Signing off a new LC column, and embeds the SOP again. Q08 passes, Q12 still refuses. All thirteen pass.

He writes the settings record, settings.md, next to the test: the AnythingLLM version, the model names, every retrieval and chat setting the workspace shows (the list is in Details for experts), the instruction and the refusal text, each with the date he last changed it. Then the README, the page a colleague follows to run the check:

The test that matters comes three weeks later. Mira adds a new SOP for phospho-enrichment and runs the check from the README, with Jonas out of the room. Q03 fails with a search miss: the new SOP also has a ratio table, and its row now comes first. She reads the printed passages, sees the pattern the README describes, gives the new table's rows their column names, embeds it again and gets PASS. Her one question for Jonas was where the API key lives; the README now says.

Where his build stops. The group's assistant runs on one PC, for one group, with SOPs the group writes. Two requests would take it past that. Colleagues want it on their own desks with personal logins, and the institute wants it to read SOPs directly from its document management system, where each group sees only its own. Both need accounts, access rights and a server someone keeps running: that's the institute's IT, and T04-L04 · Connect an Assistant to Real Documents Without Widening Anyone's Access. Jonas brings them his test set and README, which is the most valuable part of what he built.

What it taught him: the fixes took an hour. Writing thirteen questions with confirmed answers took two, and that's what made every later change safe.

Company: the policy assistant after the handbook grew

Structure it. Markus leads a customer service team of five who answer customers from a shared Claude project with their policies and the supplier handbook (T04-L02 · Curate a Document Set Your Team Can Ask). "My team quotes the assistant's answers to customers. A useful result: for our twenty most common questions it gives the current rule with the right clause, and says when our documents don't answer. My deputy Selin can check that without me."

Understand it. Since the handbook grew to sixty pages, the project shows the indicator that it now searches its files instead of reading them all. That's when answers about delivery times went wrong. The project has no retrieval settings to change, and it doesn't show the passages it found. So he changes the instruction to make the search visible: "Quote the passage you used, word for word, with file name and section." The quote tells him what the search found.

He writes twelve test questions in a spreadsheet with the columns question, expected file and section, phrase the answer must contain, quoted passage, search OK?, answer OK?, and asks them one by one. Two failures are searches: the delivery table in the handbook reads DE | 2 | 4 | 5 in the quoted passage, with no headers, and the answer confuses standard and express delivery. One is a question: "How fast do we ship to Austria?" has two answers, standard and express. Back to Structure it, like Jonas: questions with two answers get both, labelled.

Choose tools. He stays with the Claude project, which is what his company has approved. His levers are the documents and the instruction. He splits the handbook into one file per chapter, named by chapter title, and rewrites the delivery table as one line per country and service: Austria, express: 2 working days. He asks the policy lead to confirm the new file matches the old table.

Build it. He reruns all twelve questions. Eleven pass; one answer quotes the right passage and then rounds "10 working days" to "two weeks". He adds "Keep every number and unit exactly as written" to the instruction and reruns: twelve of twelve. The spreadsheet gets a tab with the settings record: plan, files with dates, the instruction text. A short README at the top says who maintains it, when to run the sheet (after every document change, and on the first Monday of the month), and what to do on a failure: stop quoting that topic from the assistant, tell Markus, and answer from the policy file until it's fixed. Selin runs the sheet the next month on her own, in 25 minutes. The only thing she changed was a question the supplier had made obsolete, and she asked the policy lead first.

The stop line is the same as in the lab. When sales asks for the assistant inside Outlook, drafting replies from SharePoint with each person's own access rights, that's a connection to company systems. Markus sends the request to IT with his test sheet attached (T14-L03 · Turn a Connector Brief Into a Routine Colleagues Can Rerun, T04-L04 · Connect an Assistant to Real Documents Without Widening Anyone's Access).

What goes wrong

Judging answers without looking at passages. Symptom: you try three prompt wordings for a wrong answer and none helps. Fix: open the passages first. If the right sentence never arrived, no instruction can use it. Fix the document or the search.

Tables that lose their headers. Symptom: the answer has the right numbers in the wrong places. Fix: rewrite table rows so each carries its column names, or keep the table short enough to stay in one passage. Add a table question to the test set.

Several changes at once. Symptom: the score goes up after new chunking, more snippets and a new model, and nobody knows which one helped, or what else it changed. Fix: one change, then all questions again, compared one by one.

A higher total that hides a new fault. Symptom: 12 of 13 before and after, and a question that used to refuse now invents. Fix: list which questions changed, in both directions, after every change.

A test that bends to pass. Symptom: expected answers quietly change after a failure. Fix: only the person who confirms the documents may change an expected answer, and the change is written down with its reason.

No question the documents can't answer. Symptom: the assistant answers everything, including questions nobody documented. Fix: keep at least one absent question in the set, and check it after every setting change. More passages make invented answers more likely.

A changed document that still gives old answers. Symptom: you edited the SOP, the answer stays the same. Fix: remove and re-add the document so it's embedded again, then rerun the check. A changed chunk size also applies only to documents embedded afterwards.

Do it yourself (3 hours, two sessions)

1. Structure it (15 min). Write two sentences: who acts on the assistant's answers, and what a correct answer must do, including what happens when the documents don't say.
2. Understand it (60 min). Collect ten to fifteen real questions. For each, write the document, the answering sentence and the phrase a correct answer must contain; include a table, a may or except, an identifier and one absent question. Get the expected answers confirmed. Ask them all, read the passages behind every answer, and label each failure search, answer or question. If a question has two answers, rewrite your sentences from step 1.
3. Choose tools (15 min). Decide what you can change: documents, instruction, which retrieval settings. Choose between a scripted check (if your tool has an API) and a spreadsheet run by hand.
4. Build it: one change at a time (45 min). Fix documents first. After each change, rerun every question and write down what moved in both directions. Keep a change only if nothing broke.
5. Build it: record and README (30 min). Write the settings record and a README: who maintains it, how to run the check, when to run it, and what to do on each kind of failure.
6. Build it: the rerun (15 min). Ask your colleague to change or add one document and run the check from the README while you stay out of the room. Fix the README wherever they got stuck.

Exit check

What you hand in: a test folder or sheet for your group's document assistant with:

• ten to fifteen test questions, each with the expected document, passage phrase and answer phrase, confirmed by the person responsible for the documents, including one the documents can't answer;
• the settings record: tool and version, model, retrieval settings you can see, instruction and refusal text, with dates;
• a before-and-after result for one change, listing which questions it fixed and which it broke;
• a README with the maintainer, a deputy, when to run the check and what to do on a search miss, a wrong answer and an ambiguous question;
• your colleague's rerun after they changed or added a document.

It passes when your colleague's rerun ends in PASS, or every failure is labelled search, answer or question with its next step, and they needed nothing beyond the README.

Rule to remember

Read the passages before you fix anything, and change one thing at a time.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T04-L02 · Curate a Document Set Your Team Can Ask: the rule and register that decide what goes in.
• T03-L03 · Put a Test Set Under the Prompt Your Team Depends On: fixed test cases and one change at a time, for prompts.
• T04-L04 · Connect an Assistant to Real Documents Without Widening Anyone's Access: connecting live document systems with each person's access rights.
• T18-L03 · Build a Cell-Counting Pipeline Your Group Can Rerun and Check: the same known-good check and README for an image pipeline.
• T14-L03 · Turn a Connector Brief Into a Routine Colleagues Can Rerun: when the assistant has to reach your other systems.

Tools

• AnythingLLM: Taught. The local document assistant in the lab example, with visible passages, retrieval settings and an API.
• Claude: Taught. The shared project in the company example.
• OpenCode: Catalogued. Wrote the first draft of the check script.
• Open WebUI: Catalogued. The alternative with comparable retrieval settings, for use from several desks.
• Heidelberg AI Desk: Catalogued. The route to trying Open WebUI on your own machine.
• Weaviate: Catalogued. A vector database with hybrid search, for builds that outgrow a desktop app.