Retrieval that cites and doesn't lie

Your colleague does not need a more impressive answer. They need the right passage before the next decision.

Perhaps your document assistant helps someone find a maintenance instruction, explain a service condition, or locate the method behind a research note. Last week you answered beside them and caught the mistakes. This week they will use it without you. A source badge is no longer enough: they must be able to open the relevant passage, see its limits, and know what to do when the assistant cannot support an answer.

Keep the project you chose. This book helps you turn its retrieval step into something another person can depend on. The result is a useful cited-answer service with a repeatable way to diagnose failures, not a new tracker or a compulsory lab exercise. The title describes an ambition, not a guarantee: retrieval can fail, and fluent answers can still invent. Your build must make those limits visible.

1. Start where your colleague will work

Open the document-question workspace or search application you already use for your project. Choose one recurring question a colleague actually needs answered. Ask it, open its citation, and locate the sentence, table row or section that supports the answer. If the answer says "within five days," find the unit, starting event and exception, not just the number five.

That is the user-facing loop you are building:

Colleague's question -> eligible sources -> ranked passages -> supported answer -> openable evidence -> use or fallback.

The builder needs a second view underneath it: the ordered search results and the exact context supplied to the answer stage. A citation panel may show only cited results, not all retrieved results or all text the model received. Do not treat it as a complete trace unless your tool documents that behavior. If your approved application hides these stages, ask its maintainer for a search-only result or existing trace export. You can still inspect an answer manually, but you cannot yet claim to have isolated retrieval from generation.

A concrete interface: AnythingLLM

If your project already uses AnythingLLM, its official document-chat guide places retrieval controls in the workspace settings. In the AnythingLLM application, hover over the test workspace and select its Gear icon. Under Vector Database Settings, locate Max Context Snippets and Document similarity threshold. Record their current values before changing anything. These are concrete levers: how many snippets may enter context, and which low-scoring matches are excluded.

Where supported, Search Preference offers Accuracy Optimized reranking. The checked guide limits this option to LanceDB and says first use downloads a reranking model. It is optional, not a reason to change databases or install a model for this book. Check availability and approval before enabling it; measure its effect instead of assuming "accuracy" in a label proves improvement.

Keep your baseline in the existing workspace and use an approved test workspace or index for the candidate. Do not experiment in the shared workspace while a colleague is relying on it. If your version lacks a documented copy operation, retain the original and have the maintainer provision a separate test scope through your established path. Do not invent a "Clone" button.

Also confirm what you are testing. AnythingLLM distinguishes chat attachments, which supply full document text within context limits, from embedded workspace documents, which use retrieved chunks. Pinned documents use full-text insertion and are excluded from the normal RAG process. An attached or pinned answer is not evidence that chunk retrieval improved. In multi-user mode, embedding makes a document available to users with workspace access; use only public, synthetic or explicitly approved material for that audience.

You do not need AnythingLLM if your current tool already exposes the required evidence. Keep your tool; reproduce the same observable stages.

2. Half right, with no visible pattern

Your team asks twenty questions of a new document assistant. Half of the answers are excellent. The rest are confidently wrong. One misses a protocol table, another turns "may" into "must," and a third cites a relevant section without answering what was asked. Every failure looks like the same thing in the chat window: a polished answer with a source badge.

The team proposes three fixes at once: make the prompt stricter, switch models, and increase the number of retrieved passages. That may move the score, but it destroys the diagnosis. If the required passage never reached the model, prompt changes cannot recover it. If the correct passage arrived and the answer overstated it, larger retrieval will not repair generation. If the question combines two meanings, neither stage has a stable target.

Treat retrieval and answering as separate components with separate tests. Freeze ten questions and their expected evidence. Capture the passages returned before any model writes. Then score the answer against those passages and against the question. Change one retrieval lever, rerun the same cases, and compare case by case. The pattern becomes visible, and the next repair has a named owner.

The ten cases support your project; they do not replace it. Choose questions from the recurring work, including awkward cases, rather than forcing the work into a convenient benchmark. The person waiting for an answer should recognize why these questions matter.

3. What you will build on

By the end, you should be able to diagnose whether a wrong answer began in retrieval, generation or the question contract; improve one retrieval behavior without changing the model; and give a colleague a service they can use and recover without your narration.

A knowledge base your team can ask establishes which documents belong in the corpus and who keeps them current. Continue that corpus and its inclusion rule. Here, the question is whether the system selects the right evidence and faithfully uses it. Test-driven prompting supplies fixed cases and one-change comparisons. Permission-aware retrieval takes the next step into live connectors and per-user access. Do not build those integrations merely to complete this book; existing permissions still apply to every test and handover.

You need an approved test environment with inspectable search results, permission to retain questions and evidence, and a colleague or source owner who can check expected answers. Python 3.10 or newer is needed only if you choose the optional local harness in section 6. The main project can use your existing interface and report format.

Keep the test read-only. It must not send messages, change business records, approve terms or make scientific decisions. Save evidence within its approved access boundary; source text in a test report is still source text, not harmless metadata.

4. Ask three questions in order

A retrieval-augmented generation system has at least two visible stages: search returns candidate passages, then a model uses those passages to compose an answer. Evaluation adds a third concern: whether the requested answer is clear and useful.

Did retrieval fetch the right passage?

Write the expected source and evidence location before the run. A file name alone is too broad when a document contains current rules, old rules and exceptions. Record the section, page, clause or table row and the exact passage needed. Then ask whether that evidence appears in the first k results, preserving their returned order.

The optional harness uses hit@1: 1 when the expected source is first, otherwise 0. Its source IDs each identify one tiny passage. In your project, a top-ranked file is not a hit if its returned chunk lacks the required evidence. For a question needing two passages, define whether both must appear and record coverage explicitly. Recall at k asks what fraction of the required evidence was retrieved within that limit; it is not interchangeable with "at least one useful result."

Is the answer grounded in the fetched passages?

Every factual clause, number, scope word and modal verb must be supported by the text supplied to the answer stage. "May" does not support "must." A citation to the correct file does not support a sentence assembled from an unrelated section.

Do not score groundedness against the whole corpus. The answer stage did not receive the whole corpus. Preserve both the search results and the final supplied context if reranking, truncation or context assembly changes them. Search can find the right passage and still lose it before generation. In that case, the context assembly is the repair point, not necessarily the language model.

Also retain wrong passages: without them, a retrieval failure becomes impossible to reproduce. An answer that accurately quotes an irrelevant archive can be grounded and still be useless.

Does the answer address the question?

A supported fact can be non-responsive. If the user asks who approves an exception and the answer gives the exception's duration, retrieval and quotation may look plausible while the task remains unanswered. Define what the colleague needs to proceed: an owner, a deadline with its starting event, or a comparison with both conditions intact.

For the narrow synthetic facts below, exact matching is a useful deterministic grader. Real summaries often need a written entailment rubric and a human reviewer because several phrasings may be valid. A model-based grader can assist, but it can miss negation, scope, tables or shared model biases. Calibrate it against human-marked cases and never hide the raw evidence behind its aggregate.

What if the answer is not in the corpus?

Include an absent-fact question from your project. Define its expected behavior before testing: for example, "The approved sources do not state the renewal date; open the agreement register or ask its owner." Do not assign it an imaginary expected passage and count every run as a retrieval miss. Mark positive-evidence retrieval as not applicable, retain what the search returned, and score whether the answer correctly withheld the unsupported claim and gave the agreed fallback. Report that case separately from the denominator for answerable retrieval cases.

Similarly, two conflicting current sources are not an invitation to average their numbers. The useful answer names the conflict and routes the colleague to the source owner. "Not established in these sources" is more precise than "does not exist." An empty search result cannot prove universal absence.

5. Know which retrieval lever changes what

Chunk size determines how much text travels as one searchable unit. Tiny chunks can lose the subject, heading, units, table header or exception that gives a sentence meaning. Huge chunks can mix topics, dilute the matching passage and consume the context budget. There is no universal correct size; choose candidate sizes from document structure and score them on your questions.

Overlap repeats a boundary region in adjacent chunks. It can preserve a sentence or definition split at a fixed boundary. Too much overlap creates near-duplicates that crowd the top results and make one source look like several independent sources. Measure both misses and duplicate occupancy.

Structure-aware chunking keeps a heading, list label or table header with the content it governs. Store stable source ID, section path, page or clause, version and status beside each chunk. Do not infer those fields from an answer after the fact. A readable citation needs enough of that locator for the colleague to find the evidence, even when a link opens the document rather than the precise passage.

Metadata filters remove ineligible candidates before or during ranking. A filter such as status = current or document_type = protocol can prevent a relevant but superseded source from winning. A filter is only as reliable as its metadata. A relevance or currency filter is not a substitute for permission enforcement; that boundary continues in T04-L04.

Hybrid search combines lexical matching with semantic similarity. Lexical search protects exact identifiers, error codes, clause numbers, dates and unusual names. Semantic search helps when the question and source use different wording. Record the combination rule or weight; "hybrid" is not a reproducible configuration by itself.

Reranking takes a candidate set from the first search and orders it with a second relevance calculation. It can move a better passage into the limited context window, but it cannot recover a passage excluded by the corpus, filter or first-stage candidate limit. Evaluate first-stage recall and final ranking separately when reranking is enabled.

Context limits and thresholds decide how many of those matches survive. Increasing k may recover evidence, but may also add contradictions, near-duplicates or text that is later truncated. Lowering a similarity threshold may admit a missing passage and more irrelevant ones. Neither knob is a truth control. Inspect the changed cases and the actual supplied context.

Change one lever per comparison. Keep the corpus revision, questions, expected evidence, answer stage, model route and graders fixed. Hold k fixed unless it is the declared variable; if you change it, report both values and do not present the comparison as a ranking-only improvement. A chunking change requires new indexed chunks, not merely a new setting displayed beside an old index. Preserve the baseline index until the candidate has been checked.

Get a search-only result without adding a new tool

If your project already uses the OpenAI API with an approved vector store and initialized Python client, the official Retrieval guide documents a direct search call. In the existing development console or notebook, replace the placeholder with that test store's ID and use one of your fixed questions:

results = client.vector_stores.search(
    vector_store_id="YOUR_APPROVED_TEST_STORE_ID",
    query="YOUR_FIXED_PROJECT_QUESTION",
)
print(results)

This calls search, not an answer model. Inspect the returned source identifiers, scores and content; retain their order in your approved evidence location. A score ranks a match, not the truth of an answer. Use the same search options for the baseline and candidate except for the declared change. This snippet assumes your existing client, credentials and store; it is not an installation exercise, and no key belongs in the report. The call is documented here, not claimed as a credentialed run.

For another retrieval product, use its existing equivalent. If only final answers are available, record that limitation rather than filling the retrieval column from a guess. The local harness below lets you understand the separation without granting yourself access to a production system.

6. Optional worked example: one heading-aware fix, two domains

Continue directly with your project in section 7 if you already have usable traces. Otherwise, choose either of these fictional examples to see a complete comparison. Neither is a required replacement project. The Lab corpus contains fictional Northstar protocols; the Company corpus contains fictional Harbor contracts. Each has ten questions, four archive distractors, one deliberately overconfident answer behavior and no real data.

The baseline imitates chunks split from their section headings: only each body is searchable. The candidate prepends the heading to the unchanged body. Search, top k, questions, expected sources, answer function and graders remain fixed. This tiny lexical retriever is an inspectable teaching harness, not a production search engine and not a benchmark for AnythingLLM, Onyx, Weaviate or any model.

In your file manager, create a new empty folder named retrieval-practice in an approved local location. In a text editor, create retrieval_eval.py inside it with the code below. In a terminal opened in that folder, confirm python --version or python3 --version reports Python 3.10 or newer. If Python is unavailable, read the example and use your existing project tool; installing it is not a prerequisite for the main path.

The script writes retrieval-evaluation.md in the terminal's current folder and replaces that file on every run. Keep this folder separate from your real report. Save any reviewed output under a distinct run name before rerunning; the Lab and Company runs use the same output filename.

from pathlib import Path
import re
import sys

MODE = sys.argv[1] if len(sys.argv) > 1 else "lab"
if MODE not in {"lab", "company"}:
    raise SystemExit("usage: python retrieval_eval.py [lab|company]")

DATA = {
    "lab": [
        ("LAB-01", "What is the NS-7 calibration interval?", "NS-7 calibration interval",
         "Every 14 days.", ["14", "days"]),
        ("LAB-02", "At what temperature must reagent lot R-17 be stored?", "R-17 storage temperature",
         "Store reagent lot R-17 at 4 C.", ["R", "17", "4"]),
        ("LAB-03", "What signal threshold accepts assay QX-41?", "QX-41 acceptance threshold",
         "Accept assay QX-41 when signal is at least 0.82.", ["QX", "41", "0", "82"]),
        ("LAB-04", "How many washes does protocol P-204 use?", "P-204 wash count",
         "Protocol P-204 uses three washes.", ["P", "204", "three"]),
        ("LAB-05", "What rotor speed does centrifuge CF-9 use?", "CF-9 centrifuge rotor speed",
         "Use 1200 rpm.", ["1200", "rpm"]),
        ("LAB-06", "How many blank controls may fail?", "Blank-control failure limit",
         "At most two blank controls may fail.", ["two", "blank", "controls"]),
        ("LAB-07", "How many times may sample S-88 be thawed?", "S-88 thaw limit",
         "Sample S-88 may be thawed once.", ["S", "88", "thawed"]),
        ("LAB-08", "How long should plate B-12 incubate?", "B-12 incubation duration",
         "Incubate plate B-12 for 30 minutes.", ["B", "12", "30"]),
        ("LAB-09", "Which wavelength is configured for reader RD-5?", "RD-5 reader wavelength",
         "Set it to 450 nm.", ["450", "nm"]),
        ("LAB-10", "Who owns NS-7 control-drift escalation?", "NS-7 control-drift escalation owner",
         "The assay supervisor.", ["assay", "supervisor"]),
    ],
    "company": [
        ("CO-01", "What is contract CT-7's renewal notice period?", "Contract CT-7 renewal notice period",
         "Give 30 calendar days notice.", ["30", "days"]),
        ("CO-02", "When must order O-17 be dispatched?", "Order O-17 dispatch window",
         "Dispatch order O-17 within 5 business days.", ["O", "17", "5"]),
        ("CO-03", "What is contract CT-41's liability cap?", "Contract CT-41 liability cap",
         "Contract CT-41 caps liability at EUR 80,000.", ["CT", "41", "80", "000"]),
        ("CO-04", "When is invoice S-204 due?", "Schedule S-204 invoice due date",
         "Invoice S-204 is due 20 calendar days after receipt.", ["S", "204", "20"]),
        ("CO-05", "What insurance minimum applies to supplier SP-9?", "Supplier SP-9 insurance minimum",
         "Maintain EUR 2 million coverage.", ["2", "million"]),
        ("CO-06", "How many service credits may be claimed?", "Service-credit claim limit",
         "At most two service credits can be claimed.", ["two", "service", "credits"]),
        ("CO-07", "How many times may order O-88 be cancelled?", "Order O-88 cancellation option",
         "Order O-88 may be cancelled once.", ["O", "88", "cancelled"]),
        ("CO-08", "How quickly must contract B-12 incidents receive a response?", "Contract B-12 response time",
         "Respond to contract B-12 incidents within 30 minutes.", ["B", "12", "30"]),
        ("CO-09", "Which law governs schedule RD-5?", "Schedule RD-5 governing law",
         "It is governed by the laws of Ireland.", ["Ireland"]),
        ("CO-10", "Who owns contract CT-7 price escalation?", "Contract CT-7 price escalation owner",
         "The commercial director.", ["commercial", "director"]),
    ],
}

DISTRACTORS = {
    "lab": [
        ("LAB-D1", "Archive note", "Calibration frequency appears here, but no current interval. Answer: Archive only."),
        ("LAB-D2", "Maintenance note", "This mentions centrifuge rotor speed but no CF-9 value. Answer: Not specified."),
        ("LAB-D3", "Optics note", "Reader wavelength appears without an RD-5 setting. Answer: Not specified."),
        ("LAB-D4", "Escalation archive", "NS-7 control-drift escalation is obsolete. Answer: Archive only."),
    ],
    "company": [
        ("CO-D1", "Archive note", "Renewal notice appears here, but no current period. Answer: Archive only."),
        ("CO-D2", "Insurance note", "Supplier insurance minimum is discussed without SP-9. Answer: Not specified."),
        ("CO-D3", "Legal note", "Schedule law is mentioned without RD-5. Answer: Not specified."),
        ("CO-D4", "Escalation archive", "Contract price escalation is obsolete. Answer: Archive only."),
    ],
}

STOP = {"a", "an", "at", "be", "by", "does", "for", "how", "is", "it", "of", "the", "to", "what", "when", "which", "who"}

def tokens(text):
    return {word for word in re.findall(r"[a-z0-9]+", text.lower()) if word not in STOP}

def corpus(include_heading):
    chunks = []
    for source, _, heading, expected, _ in DATA[MODE]:
        body = f"Controlled synthetic fact. Answer: {expected}"
        searchable = f"{heading}. {body}" if include_heading else body
        chunks.append({"source": source, "body": body, "searchable": searchable})
    for source, heading, body in DISTRACTORS[MODE]:
        searchable = f"{heading}. {body}" if include_heading else body
        chunks.append({"source": source, "body": body, "searchable": searchable})
    return chunks

def retrieve(query, chunks):
    query_terms = tokens(query)
    ranked = sorted(chunks, key=lambda chunk: (-len(query_terms & tokens(chunk["searchable"])), chunk["source"]))
    return ranked[0]  # hit@1: top k is deliberately fixed at one

def generate(chunk):
    answer = chunk["body"].split("Answer: ", 1)[1]
    return answer.replace(" may be ", " must be ")  # seeded generation defect

def evaluate(label, include_heading):
    rows = []
    chunks = corpus(include_heading)
    for case_id, question, _, expected_answer, address_terms in DATA[MODE]:
        chunk = retrieve(question, chunks)
        answer = generate(chunk)
        retrieval = chunk["source"] == case_id
        grounded = answer in chunk["body"]
        addresses = tokens(" ".join(address_terms)) <= tokens(answer)
        rows.append({
            "run": label,
            "case": case_id,
            "question": question,
            "expected_source": case_id,
            "expected_passage": f"Controlled synthetic fact. Answer: {expected_answer}",
            "top_source": chunk["source"],
            "retrieved_passage": chunk["body"],
            "retrieval": retrieval,
            "grounded": grounded,
            "addresses": addresses,
            "answer": answer,
        })
    return rows

def totals(rows):
    return tuple(sum(row[field] for row in rows) for field in ("retrieval", "grounded", "addresses"))

def case_changes(before_rows, after_rows):
    fixed = [
        after_row["case"] for before_row, after_row in zip(before_rows, after_rows)
        if not before_row["retrieval"] and after_row["retrieval"]
    ]
    regressed = [
        after_row["case"] for before_row, after_row in zip(before_rows, after_rows)
        if before_row["retrieval"] and not after_row["retrieval"]
    ]
    blocking = [
        row["case"] for row in after_rows
        if not (row["retrieval"] and row["grounded"] and row["addresses"])
    ]
    return fixed, regressed, blocking

def cell(value):
    return str(value).replace("|", "\\|").replace("\n", " ")

before = evaluate("before", include_heading=False)
after = evaluate("after", include_heading=True)  # the only changed retrieval setting
assert totals(before) == (6, 9, 6), totals(before)
assert totals(after) == (10, 9, 10), totals(after)
fixed, regressed, blocking = case_changes(before, after)

lines = [
    f"# Retrieval evaluation: {MODE}",
    "",
    "- Corpus: synthetic fixture v1",
    "- Top k: 1",
    "- Fixed answer stage: extract `Answer:` and replace `may be` with `must be`",
    "- Change: prepend each section heading to its body chunk",
    "",
    "| Run | Case | Question | Expected source | Expected passage | Top source | Retrieved passage | Retrieval | Grounded | Addresses | Answer |",
    "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",
]
for row in before + after:
    values = [
        row["run"], row["case"], row["question"], row["expected_source"],
        row["expected_passage"], row["top_source"], row["retrieved_passage"],
        row["retrieval"], row["grounded"], row["addresses"], row["answer"],
    ]
    lines.append("| " + " | ".join(cell(value) for value in values) + " |")

before_total = totals(before)
after_total = totals(after)
lines += [
    "",
    "| Run | Retrieval hit@1 | Grounded in retrieved passage | Addresses |",
    "| --- | ---: | ---: | ---: |",
    f"| Before | {before_total[0]}/10 | {before_total[1]}/10 | {before_total[2]}/10 |",
    f"| After | {after_total[0]}/10 | {after_total[1]}/10 | {after_total[2]}/10 |",
    "",
    f"Fixed retrieval cases: {', '.join(fixed) if fixed else 'none'}.",
    f"Regressed retrieval cases: {', '.join(regressed) if regressed else 'none'}.",
    f"Remaining blocking cases: {', '.join(blocking) if blocking else 'none'}.",
    "Human spot-check: PENDING - a reviewer must record one pass, one retrieval miss from the before run, and one groundedness failure.",
    "Decision: keep heading-aware chunks; the generation owner must fix the may/must defect, rerun, and obtain the human spot-check before release.",
]
Path("retrieval-evaluation.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
print("\n".join(lines[-11:]))

Run one framing in that terminal:

python retrieval_eval.py lab
# or, on systems where Python 3 is named python3:
python3 retrieval_eval.py company

The Lab command should print the following. These are deterministic fixture expectations, not measurements of your project or a model:

| Run | Retrieval hit@1 | Grounded in retrieved passage | Addresses |
| --- | ---: | ---: | ---: |
| Before | 6/10 | 9/10 | 6/10 |
| After | 10/10 | 9/10 | 10/10 |

Fixed retrieval cases: LAB-01, LAB-05, LAB-09, LAB-10.
Regressed retrieval cases: none.
Remaining blocking cases: LAB-07.
Human spot-check: PENDING - a reviewer must record one pass, one retrieval miss from the before run, and one groundedness failure.
Decision: keep heading-aware chunks; the generation owner must fix the may/must defect, rerun, and obtain the human spot-check before release.

The Company run prints the parallel CO-01, CO-05, CO-09, CO-10, and CO-07 case IDs. Open the newly written retrieval-evaluation.md in your editor and inspect the expected and retrieved passages. In a separately saved review copy, replace the pending spot-check line only after a person has checked the rows; record their name or role, date, three case IDs and decision. The script itself does not perform that review.

Read the change, not just the score

Before the fix, LAB-01, LAB-05, LAB-09 and LAB-10 lose their subject because their bodies contain only values such as Every 14 days or Use 1200 rpm. Archive chunks share more query terms, so retrieval chooses them. Prepending headings restores NS-7 calibration interval, CF-9 centrifuge rotor speed, RD-5 reader wavelength and NS-7 control-drift escalation owner. Retrieval rises from 6/10 to 10/10 without touching the answer stage.

Groundedness remains 9/10. A wrong retrieved passage can support an answer that does not address the question, so support must not be scored as correctness. LAB-07 demonstrates the other direction: the correct passage arrives, but the answer function changes may be thawed once to must be thawed once. It addresses the thaw question, yet it is not grounded. The next experiment belongs to generation, not chunking. A source owner still decides real protocol meaning; this example does not approve a scientific procedure.

The Company run is parallel. Heading loss hides the renewal period, insurance minimum, governing law and escalation owner in CO-01, CO-05, CO-09 and CO-10. Heading-aware chunks lift retrieval to 10/10, while CO-07 still turns an option to cancel into a requirement. A contract owner, not the retrieval builder, decides whether a real clause is authoritative or how it applies.

Notice the deliberate limits of the graders. grounded checks a literal substring, not general entailment. addresses checks a set of words, not whether the answer expresses the right relationship. The top-1 retriever always chooses something, even with no meaningful match. The fixture has no absent-fact case, no multi-passage answer and no permission test. Use it to understand the diagnostic split, not to certify a real service.

The report recommends keeping the retrieval improvement but blocks release of the faulty answerer. These are different decisions. If you repair the seeded may/must behavior as a second experiment, preserve this comparison first and revise the groundedness expectation for that new run. The supplied 9/10 assertions deliberately describe the unfixed generator; do not silently alter them to make the original experiment look successful.

7. Improve your own recurring result

Return to your chosen project. State its service promise in the language of work: "A colleague can find the current instruction and its exception before preparing the next run," or "A colleague can answer a routine service question with the applicable clause attached." Those are optional patterns, not assigned scenarios. Define who uses the answer, what they do next, and where they go when evidence is insufficient.

Establish the baseline

In your existing project notes or approved shared document, create a clearly named baseline section. Record the corpus and configuration revision, date and test scope. Record the settings that exist in your system: chunker, embedder, lexical/semantic weights, filters, reranker, candidate count, top k, model route and prompt revision. Mark an unavailable setting as unknown rather than inventing its default.

Write ten fixed questions before changing settings. Include an exact identifier, a heading-dependent fact, a table or list, a qualifier such as may or except, an absent fact and ordinary recurring work. Let one case cover several risks when natural. Ask the source owner to confirm the expected passage and interpretation. If an expectation is later found wrong, create a revised set and rerun both configurations; do not quietly improve only the candidate's answer key.

For each answerable case, retain the question, expected passage and location, ordered returned chunks, exact supplied context, generated answer and three separate judgments. Record failures as well as passes. For the absent case, use the no-evidence rule in section 4 and show the denominator separately. A report in Markdown, a shared document or your existing evaluation interface is sufficient; a CSV conversion adds nothing by itself.

Run retrieval without generation where your tool supports it. Then run the unchanged answer stage over those captured passages. If the application always performs a fresh search when answering, preserve that run's actual context too. Do not claim a replay used frozen passages when it did not. Keep model route, prompt, conversation state and sampling settings fixed where available; a real model may still vary, so label a single comparison as bounded evidence rather than a guarantee.

Choose the smallest repair your traces justify

Group misses by cause: absent source, filter exclusion, heading loss, boundary split, identifier mismatch, semantic mismatch or ranking loss. Pick one observed group, then write one hypothesis. "The requested identifier is missing from value-only chunks; keeping the heading should restore it" points to a chunking experiment. "The answer changes may to must" does not.

For an AnythingLLM threshold experiment, return to the test workspace's Gear > Vector Database Settings > Document similarity threshold. Record the exact existing value, change only that control to your chosen candidate value, apply the change through the installed interface, and reopen settings to confirm it persisted. Keep Max Context Snippets, Search Preference, documents and answer settings unchanged. Capture the new traces for the same questions. If you cannot inspect whether an excluded passage now appears, you can compare answer behavior but cannot attribute improvement to that hypothesized retrieval repair.

For a heading repair, change the test ingestion rule or approved source preparation so each indexed passage carries its governing heading. Rebuild only the candidate index through your existing ingestion path, confirm it is ready, then search it. Do not overwrite the authoritative document to make the benchmark easier. Keep the corpus content and expected evidence fixed; the changed representation is what you are testing.

Rerun all ten cases. List fixed, regressed, unchanged and still-blocking cases, not just totals. Inspect every moved case. A current-source filter may improve precision while revealing missing status metadata. A reranker may improve ranking but make the colleague wait too long. Preserve both outcomes and decide against the acceptance rule you wrote before the change.

Put a colleague at the controls

Give the colleague the ordinary user entry point, not the builder console. Ask them to complete one representative task without your coaching: ask the question, open the cited source, locate the supporting passage and decide whether the answer can be used. Ask them to try the agreed no-evidence route as well. This tests the dependency that the scores alone cannot show.

Can they read the citation's title, version and location? Can they open it with their ordinary permissions and navigate with their usual keyboard or assistive technology? Do not provide evidence only as a screenshot or mark success only by color. If the citation opens for an administrator but not for the colleague, the handover is not ready. Fix the authorized evidence path without widening access just to make a test pass.

Have a colleague or source owner independently inspect at least three diagnostic rows: one pass, one retrieval miss from the baseline and one groundedness failure if present. If your project has no observed groundedness failure, say so; use the explicitly synthetic fixture to discuss that failure mode, not to fabricate a project incident. Record the actual cases reviewed and any correction. A proposed check is not a completed check.

Leave a recovery path, not a warning label

In the same handover, write where the colleague starts, the supported question boundary, and the source-first fallback. For example: if an answer lacks support or sources conflict, stop using that answer, open the approved source register and contact its named owner. For consequential work, the assistant's answer is not an approval.

For the maintainer, name the known-good workspace or index and its saved configuration, the candidate revision, and the exact established action used to return to the known-good route. If your platform lacks a routing switch, the fallback may be the unchanged baseline workspace or manual source lookup, with colleagues explicitly told which to use. Do not delete either index as your rollback strategy. Retire temporary traces and candidate indexes only after the decision, required evidence retention and recovery needs are satisfied.

A colleague reporting a bad answer should be able to provide the question, time, workspace/version and source locator in an approved channel. They should not have to paste restricted passages into an unrestricted ticket. The maintainer can reproduce the failure from retained traces within the correct boundary.

8. What goes wrong

The prompt is tuned to fix a retrieval miss

Symptom: the expected source is absent from the captured results, but the team adds "use the correct document" to the answer prompt.

Fix: inspect corpus membership, query terms, chunk boundaries, filters, candidate count and ranking first. A model cannot quote a passage it never received. If the passage was found but dropped from supplied context, repair that handoff instead.

A table row is split from its header

Symptom: retrieval returns R-17 | 4 C | 48 h, but neither model nor reviewer can tell which column is temperature or duration.

Fix: keep the table title and column headers with each row group, preserve page and table identifiers, and add a question whose expected evidence requires the header. A beautifully cited number without its unit is still a bad answer.

Semantic search is used alone for exact identifiers

Symptom: a query for RD-5 returns conceptually similar reader documents but misses the exact schedule or instrument.

Fix: test lexical plus semantic retrieval, or an exact metadata filter, against identifier cases. Record weights and filters rather than merely naming the mode "hybrid."

Improvements are anecdotes

Symptom: three favorite questions look better after a change, with no baseline, denominator or regression list.

Fix: preserve the fixed set and rerun every case. Do not mix abstention cases into an answerable-source hit rate or hide a blocking failure in an average. Broaden coverage as the service's supported work expands; ten cases are a starting contract, not statistical proof of reliability.

A citation badge counts as groundedness

Symptom: a relevant file is cited, so a changed number, missing exception or may-to-must rewrite passes review.

Fix: compare each answer clause with the exact supplied passage. Check the citation's location and version separately. Keep source retrieval, answer support and usefulness in different columns.

Several levers change together

Symptom: chunk size, overlap, embedding model, filters, top k and reranker all change, producing a higher score with no causal diagnosis.

Fix: write one hypothesis and change one lever. If operational constraints force a bundle, label it as a bundle and do not attribute the result to one component. Preserve the known-good route while investigating.

The report passes, but the colleague cannot work

Symptom: the builder can reproduce the scores, but the colleague cannot find the workspace, open its sources or recover from an unsupported answer.

Fix: perform the ordinary-user task and fallback check in section 7. A report is evidence about a service; it is not a substitute for that service.

9. Exit check

Deliver one handover for your working retrieval service in the project's existing format. Include its user entry point, recurring purpose, supported scope, source-first fallback and known-good recovery route. Attach or link the evidence within the same approved access boundary: ten fixed questions, expected evidence, separate before-and-after retrieval and groundedness scores, addresses-question judgments, the one changed setting, held-fixed configuration, regressions and the actual colleague check.

Finish with two explicit decisions. Keep, revert or investigate describes the retrieval candidate. Ready for the stated use or blocked describes the service, including remaining generation, citation or access failures and their next owner. A sound diagnosis can be complete while the service is still blocked. Do not call the handover passing until its stated use and fallback work for the colleague and no blocking failure remains in that supported scope.

If a generation repair is needed, preserve the retrieval comparison and make it a separate run. Do not change the answer stage halfway through the before-and-after experiment. If you used only the optional fixture, you have practiced the method; you have not yet demonstrated that a colleague can depend on your chosen build.

The useful result is not "we reached ten out of ten." It is "my colleague can get the evidence they need, and when the system cannot provide it, they know where to go without being misled."

10. Rule to remember

Find out which half is broken before you fix anything. Then check that the person depending on it can actually use the repair.

11. Further reading and tools

• T04-L02: A knowledge base your team can ask establishes the curated corpus and source register this evaluation assumes.
• T03-L03: Test-driven prompting supplies the fixed-set, baseline, grader and one-change discipline.
• T04-L04: Permission-aware retrieval over real systems develops live connections and permission-boundary tests beyond this book's bounded build.
• AnythingLLM document-chat introduction (opens in a new tab): freshly checked official documentation for attachment versus embedding, workspace retrieval controls and pinning. Match controls to your installed version; documentation is not evidence that your instance exposes a complete trace.
• OpenAI Retrieval guide (opens in a new tab): freshly checked official search-call example for an existing vector-store client. Also documents source chunks, filtering and ranking; no credentialed call is claimed here.
• Weaviate hybrid search (opens in a new tab): inherited official reference for combining vector and keyword search and configuring their balance.
• Weaviate reranking (opens in a new tab): inherited official reference for reranking an initial result set.
• Onyx RAG and Search documentation (opens in a new tab): inherited official reference for search, filters and cited results, not a freshly verified interface walkthrough.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (opens in a new tab): retained original research reference describing the retrieval-plus-generation pattern.
• Tools index: catalogue entries for retrieval tools. Use the approved tool already serving your project rather than migrating to complete an example.

How this book was made

This revision retains the published manuscript's diagnostic framework, two fictional corpora and deterministic Python harness. Astra (OpenAI, openai/gpt-6-astra) revised the learner path, tool orientation and colleague handover, and checked the cited AnythingLLM and OpenAI documentation. The source record does not identify the original writer. No application installation, credentialed retrieval run, screenshot capture or classroom result is claimed. The examples teach what to inspect; only a run in your approved project can show whether your service meets its promise.