Retrieval that cites and doesn't lie

At Level 3 Builder, colleagues depend on a retrieval system you configured. A citation can point to a real document while the answer still uses the wrong passage, drops a qualifier, or answers a different question. Your job is to make those failures observable before changing the model or shipping the system.

2. Half right, with no visible pattern

Your team asks twenty questions of a new document assistant. Half of the answers are excellent. The rest are confidently wrong. One misses a protocol table, another turns "may" into "must," and a third cites a relevant section without answering what was asked. Every failure looks like the same thing in the chat window: a polished answer with a source badge.

The team proposes three fixes at once: make the prompt stricter, switch models, and increase the number of retrieved passages. That may move the score, but it destroys the diagnosis. If the required passage never reached the model, prompt changes cannot recover it. If the correct passage arrived and the answer overstated it, larger retrieval will not repair generation. If the question combines two meanings, neither stage has a stable target.

Treat retrieval and answering as separate components with separate tests. Freeze ten questions and their expected evidence. Capture the passages returned before any model writes. Then score the answer against those passages and against the question. Change one retrieval lever, rerun the same ten cases, and compare case by case. The pattern becomes visible, and the next repair has a named owner.

3. After this you can

• Diagnose whether a wrong answer began in retrieval, generation, or the question contract.
• Score retrieval and groundedness separately on a fixed ten-question set.
• Improve chunking and search without changing the language model.
• Use headings, overlap, metadata filters, hybrid search, and reranking deliberately.
• Run one reproducible before-and-after test and hand a colleague its evidence, scores, failures, and next decision.

4. Prerequisites

• T04-L02 - A knowledge base your team can ask, for a bounded, current corpus with an inclusion/exclusion rule.
• T03-L03 - Test-driven prompting, for fixed cases, observable expectations, baselines, and one-variable changes.
• Python 3.10 or newer for the local harness below. Confirm it works with python --version or python3 --version.
• An approved test environment where you can inspect retrieved passages before generation. A chat-only interface that hides retrieval traces is not enough for this exercise.
• Permission to retain test questions, chunk identifiers, retrieved text, and generated outputs.

Use only public, synthetic, or explicitly approved material. The complete fixture below is synthetic. Do not copy unpublished papers, participant data, customer contracts, credentials, production logs, or restricted document text into a personal evaluation tool. Keep this test read-only: it must not send messages, change records, approve terms, or make scientific or commercial decisions.

5. The idea in one page

Ask three questions in order

A retrieval-augmented generation system has at least two visible stages: search returns candidate passages, then a model uses those passages to compose an answer. Evaluation adds a third concern: whether the requested answer is clear and useful.

For every test case, ask these questions separately:

1. Did retrieval fetch the right passage? Write the expected source or chunk before the run. Score whether it appears in the first k results. This book uses hit@1: 1 when the expected source is first, otherwise 0. A larger system may use recall at k, but it must record k and preserve the returned order.
2. Is the answer grounded in the fetched passages? Every factual clause, number, scope word, and modal verb must be supported by retrieved text. "May" does not support "must." A citation to the correct file does not support a sentence assembled from an unrelated section.
3. Does the answer address the question? A supported fact can still be non-responsive. If the user asks who approves an exception and the answer gives the exception's duration, retrieval and quotation may both look plausible while the task remains unanswered.

The diagnostic table is more useful than one blended quality score:

Do not score groundedness against the whole corpus. The answer stage did not receive the whole corpus. Score it against the exact passages actually supplied. Also retain wrong passages: without them, a retrieval failure becomes impossible to reproduce.

For the narrow synthetic facts in this book, exact matching is a useful deterministic grader. Real summaries often need a written entailment rubric and a human reviewer because several phrasings may be valid. A model-based groundedness grader can assist, but it can also miss negation, scope, tables, or shared model biases. Calibrate it against human-marked cases and never hide the raw evidence behind its aggregate.

Know which retrieval lever changes what

Chunk size determines how much text travels as one searchable unit. Tiny chunks can lose the subject, heading, units, table header, or exception that gives a sentence meaning. Huge chunks can mix several topics, dilute the matching passage, and consume the context budget. There is no universal correct size; choose candidate sizes from document structure and score them on your questions.

Overlap repeats a boundary region in adjacent chunks. It can preserve a sentence or definition split at a fixed boundary. Too much overlap creates near-duplicates that crowd the top results and make one source look like several independent sources. Measure both misses and duplicate occupancy.

Structure-aware chunking keeps a heading, list label, or table header with the content it governs. Store stable source ID, section path, page or clause, version, and status beside each chunk. Do not infer those fields from an answer after the fact.

Metadata filters remove ineligible candidates before or during ranking. A filter such as status = current or document_type = protocol can prevent a relevant but superseded source from winning. A filter is only as reliable as its metadata. Permissions remain a separate control and are developed in T04-L04 - Permission-aware retrieval.

Hybrid search combines lexical matching with semantic similarity. Lexical search protects exact identifiers, error codes, clause numbers, dates, and unusual names. Semantic search helps when the question and source use different wording. Record the combination rule or weight; "hybrid" is not a reproducible configuration by itself.

Reranking takes a candidate set from the first search and orders it with a second relevance calculation. It can move a better passage into the limited context window, but it cannot recover a passage excluded by the corpus, filter, or first-stage candidate limit. Evaluate first-stage recall and final ranking separately when reranking is enabled.

Change one lever per comparison. Keep the corpus revision, questions, expected evidence, answer stage, model route, top k, and graders fixed. If you change chunking, embeddings, prompt, and model together, the bundle may improve but you will not know which repair worked.

6. The worked example: one heading-aware fix, two domains

The Lab and Company builds use the same test shape and the same one-line retrieval change. The Lab corpus contains fictional Northstar protocols. The Company corpus contains fictional Harbor contracts. Each has ten questions, four archive distractors, one deliberately overconfident answer behavior, and no real data.

The baseline imitates chunks split from their section headings: only each body is searchable. The candidate prepends the heading to the unchanged body. Search, top k, questions, expected sources, answer function, and graders remain fixed. This tiny lexical retriever is an inspectable teaching harness, not a production search engine and not a benchmark for AnythingLLM, Onyx, Weaviate, or any model.

Create retrieval_eval.py with the code below:

from pathlib import Path
import re
import sys

MODE = sys.argv[1] if len(sys.argv) > 1 else "lab"
if MODE not in {"lab", "company"}:
    raise SystemExit("usage: python retrieval_eval.py [lab|company]")

DATA = {
    "lab": [
        ("LAB-01", "What is the NS-7 calibration interval?", "NS-7 calibration interval",
         "Every 14 days.", ["14", "days"]),
        ("LAB-02", "At what temperature must reagent lot R-17 be stored?", "R-17 storage temperature",
         "Store reagent lot R-17 at 4 C.", ["R", "17", "4"]),
        ("LAB-03", "What signal threshold accepts assay QX-41?", "QX-41 acceptance threshold",
         "Accept assay QX-41 when signal is at least 0.82.", ["QX", "41", "0", "82"]),
        ("LAB-04", "How many washes does protocol P-204 use?", "P-204 wash count",
         "Protocol P-204 uses three washes.", ["P", "204", "three"]),
        ("LAB-05", "What rotor speed does centrifuge CF-9 use?", "CF-9 centrifuge rotor speed",
         "Use 1200 rpm.", ["1200", "rpm"]),
        ("LAB-06", "How many blank controls may fail?", "Blank-control failure limit",
         "At most two blank controls may fail.", ["two", "blank", "controls"]),
        ("LAB-07", "How many times may sample S-88 be thawed?", "S-88 thaw limit",
         "Sample S-88 may be thawed once.", ["S", "88", "thawed"]),
        ("LAB-08", "How long should plate B-12 incubate?", "B-12 incubation duration",
         "Incubate plate B-12 for 30 minutes.", ["B", "12", "30"]),
        ("LAB-09", "Which wavelength is configured for reader RD-5?", "RD-5 reader wavelength",
         "Set it to 450 nm.", ["450", "nm"]),
        ("LAB-10", "Who owns NS-7 control-drift escalation?", "NS-7 control-drift escalation owner",
         "The assay supervisor.", ["assay", "supervisor"]),
    ],
    "company": [
        ("CO-01", "What is contract CT-7's renewal notice period?", "Contract CT-7 renewal notice period",
         "Give 30 calendar days notice.", ["30", "days"]),
        ("CO-02", "When must order O-17 be dispatched?", "Order O-17 dispatch window",
         "Dispatch order O-17 within 5 business days.", ["O", "17", "5"]),
        ("CO-03", "What is contract CT-41's liability cap?", "Contract CT-41 liability cap",
         "Contract CT-41 caps liability at EUR 80,000.", ["CT", "41", "80", "000"]),
        ("CO-04", "When is invoice S-204 due?", "Schedule S-204 invoice due date",
         "Invoice S-204 is due 20 calendar days after receipt.", ["S", "204", "20"]),
        ("CO-05", "What insurance minimum applies to supplier SP-9?", "Supplier SP-9 insurance minimum",
         "Maintain EUR 2 million coverage.", ["2", "million"]),
        ("CO-06", "How many service credits may be claimed?", "Service-credit claim limit",
         "At most two service credits can be claimed.", ["two", "service", "credits"]),
        ("CO-07", "How many times may order O-88 be cancelled?", "Order O-88 cancellation option",
         "Order O-88 may be cancelled once.", ["O", "88", "cancelled"]),
        ("CO-08", "How quickly must contract B-12 incidents receive a response?", "Contract B-12 response time",
         "Respond to contract B-12 incidents within 30 minutes.", ["B", "12", "30"]),
        ("CO-09", "Which law governs schedule RD-5?", "Schedule RD-5 governing law",
         "It is governed by the laws of Ireland.", ["Ireland"]),
        ("CO-10", "Who owns contract CT-7 price escalation?", "Contract CT-7 price escalation owner",
         "The commercial director.", ["commercial", "director"]),
    ],
}

DISTRACTORS = {
    "lab": [
        ("LAB-D1", "Archive note", "Calibration frequency appears here, but no current interval. Answer: Archive only."),
        ("LAB-D2", "Maintenance note", "This mentions centrifuge rotor speed but no CF-9 value. Answer: Not specified."),
        ("LAB-D3", "Optics note", "Reader wavelength appears without an RD-5 setting. Answer: Not specified."),
        ("LAB-D4", "Escalation archive", "NS-7 control-drift escalation is obsolete. Answer: Archive only."),
    ],
    "company": [
        ("CO-D1", "Archive note", "Renewal notice appears here, but no current period. Answer: Archive only."),
        ("CO-D2", "Insurance note", "Supplier insurance minimum is discussed without SP-9. Answer: Not specified."),
        ("CO-D3", "Legal note", "Schedule law is mentioned without RD-5. Answer: Not specified."),
        ("CO-D4", "Escalation archive", "Contract price escalation is obsolete. Answer: Archive only."),
    ],
}

STOP = {"a", "an", "at", "be", "by", "does", "for", "how", "is", "it", "of", "the", "to", "what", "when", "which", "who"}

def tokens(text):
    return {word for word in re.findall(r"[a-z0-9]+", text.lower()) if word not in STOP}

def corpus(include_heading):
    chunks = []
    for source, _, heading, expected, _ in DATA[MODE]:
        body = f"Controlled synthetic fact. Answer: {expected}"
        searchable = f"{heading}. {body}" if include_heading else body
        chunks.append({"source": source, "body": body, "searchable": searchable})
    for source, heading, body in DISTRACTORS[MODE]:
        searchable = f"{heading}. {body}" if include_heading else body
        chunks.append({"source": source, "body": body, "searchable": searchable})
    return chunks

def retrieve(query, chunks):
    query_terms = tokens(query)
    ranked = sorted(chunks, key=lambda chunk: (-len(query_terms & tokens(chunk["searchable"])), chunk["source"]))
    return ranked[0]  # hit@1: top k is deliberately fixed at one

def generate(chunk):
    answer = chunk["body"].split("Answer: ", 1)[1]
    return answer.replace(" may be ", " must be ")  # seeded generation defect

def evaluate(label, include_heading):
    rows = []
    chunks = corpus(include_heading)
    for case_id, question, _, expected_answer, address_terms in DATA[MODE]:
        chunk = retrieve(question, chunks)
        answer = generate(chunk)
        retrieval = chunk["source"] == case_id
        grounded = answer in chunk["body"]
        addresses = tokens(" ".join(address_terms)) <= tokens(answer)
        rows.append({
            "run": label,
            "case": case_id,
            "question": question,
            "expected_source": case_id,
            "expected_passage": f"Controlled synthetic fact. Answer: {expected_answer}",
            "top_source": chunk["source"],
            "retrieved_passage": chunk["body"],
            "retrieval": retrieval,
            "grounded": grounded,
            "addresses": addresses,
            "answer": answer,
        })
    return rows

def totals(rows):
    return tuple(sum(row[field] for row in rows) for field in ("retrieval", "grounded", "addresses"))

def case_changes(before_rows, after_rows):
    fixed = [
        after_row["case"] for before_row, after_row in zip(before_rows, after_rows)
        if not before_row["retrieval"] and after_row["retrieval"]
    ]
    regressed = [
        after_row["case"] for before_row, after_row in zip(before_rows, after_rows)
        if before_row["retrieval"] and not after_row["retrieval"]
    ]
    blocking = [
        row["case"] for row in after_rows
        if not (row["retrieval"] and row["grounded"] and row["addresses"])
    ]
    return fixed, regressed, blocking

def cell(value):
    return str(value).replace("|", "\\|").replace("\n", " ")

before = evaluate("before", include_heading=False)
after = evaluate("after", include_heading=True)  # the only changed retrieval setting
assert totals(before) == (6, 9, 6), totals(before)
assert totals(after) == (10, 9, 10), totals(after)
fixed, regressed, blocking = case_changes(before, after)

lines = [
    f"# Retrieval evaluation: {MODE}",
    "",
    "- Corpus: synthetic fixture v1",
    "- Top k: 1",
    "- Fixed answer stage: extract `Answer:` and replace `may be` with `must be`",
    "- Change: prepend each section heading to its body chunk",
    "",
    "| Run | Case | Question | Expected source | Expected passage | Top source | Retrieved passage | Retrieval | Grounded | Addresses | Answer |",
    "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |",
]
for row in before + after:
    values = [
        row["run"], row["case"], row["question"], row["expected_source"],
        row["expected_passage"], row["top_source"], row["retrieved_passage"],
        row["retrieval"], row["grounded"], row["addresses"], row["answer"],
    ]
    lines.append("| " + " | ".join(cell(value) for value in values) + " |")

before_total = totals(before)
after_total = totals(after)
lines += [
    "",
    "| Run | Retrieval hit@1 | Grounded in retrieved passage | Addresses |",
    "| --- | ---: | ---: | ---: |",
    f"| Before | {before_total[0]}/10 | {before_total[1]}/10 | {before_total[2]}/10 |",
    f"| After | {after_total[0]}/10 | {after_total[1]}/10 | {after_total[2]}/10 |",
    "",
    f"Fixed retrieval cases: {', '.join(fixed) if fixed else 'none'}.",
    f"Regressed retrieval cases: {', '.join(regressed) if regressed else 'none'}.",
    f"Remaining blocking cases: {', '.join(blocking) if blocking else 'none'}.",
    "Human spot-check: PENDING - a reviewer must record one pass, one retrieval miss from the before run, and one groundedness failure.",
    "Decision: keep heading-aware chunks; the generation owner must fix the may/must defect, rerun, and obtain the human spot-check before release.",
]
Path("retrieval-evaluation.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
print("\n".join(lines[-11:]))

Run one framing:

python retrieval_eval.py lab
# or, on systems where Python 3 is named python3:
python3 retrieval_eval.py company

The command writes retrieval-evaluation.md and should print:

| Run | Retrieval hit@1 | Grounded in retrieved passage | Addresses |
| --- | ---: | ---: | ---: |
| Before | 6/10 | 9/10 | 6/10 |
| After | 10/10 | 9/10 | 10/10 |

Fixed retrieval cases: LAB-01, LAB-05, LAB-09, LAB-10.
Regressed retrieval cases: none.
Remaining blocking cases: LAB-07.
Human spot-check: PENDING - a reviewer must record one pass, one retrieval miss from the before run, and one groundedness failure.
Decision: keep heading-aware chunks; the generation owner must fix the may/must defect, rerun, and obtain the human spot-check before release.

The Company run prints the parallel CO-01, CO-05, CO-09, CO-10, and CO-07 case IDs. Open retrieval-evaluation.md and inspect its expected and retrieved passages. Then replace the pending spot-check line with the reviewer's name or role, date, the three checked case IDs, and decision. Until that line is completed and the generation defect is repaired, the report is evidence for a retrieval change but not a passing release artifact.

Lab reading

Before the fix, LAB-01, LAB-05, LAB-09, and LAB-10 lose their subject because their body text contains only values such as Every 14 days or Use 1200 rpm. Archive chunks share more query terms, so retrieval chooses them. Prepending headings restores NS-7 calibration interval, CF-9 centrifuge rotor speed, RD-5 reader wavelength, and NS-7 control-drift escalation owner. Retrieval rises from 6/10 to 10/10 without touching the answer stage. Groundedness remains 9/10: a wrong retrieved passage can support an answer that does not address the question, so support must not be scored as correctness.

LAB-07 proves why that is not enough. The correct passage arrives, but the deliberately flawed answer function changes may be thawed once to must be thawed once. It addresses the thaw question, yet it is not grounded. The next experiment belongs to generation, not chunking. The assay supervisor still decides real protocol meaning; this test does not approve a scientific procedure.

Company reading

The Company run is parallel. Heading loss hides the renewal period, insurance minimum, governing law, and escalation owner in CO-01, CO-05, CO-09, and CO-10. Heading-aware chunks lift retrieval from 6/10 to 10/10, while groundedness remains 9/10. CO-07 still turns an option to cancel into a requirement to cancel. That groundedness failure blocks release even though aggregate retrieval is perfect.

The contract owner, not the retrieval builder, decides whether a clause is authoritative or how a real contract applies. The builder proves only that this version retrieves the expected synthetic passage and preserves its wording under the recorded grader.

Move the harness to a real test system

Export or log the top passages from your approved retriever and keep the same report columns. Replace the teaching retriever with those traces; do not copy production text into this script. Record the corpus revision, chunker revision, embedding model if used, lexical and semantic weights, filters, reranker, candidate count, top k, model route, prompt revision, and date. Run retrieval first with generation disabled. Only then run the unchanged answer stage over the captured passages.

When comparing a real system, do not require all settings to be "better." Require the intended metric to improve without a blocking regression. A metadata filter may increase current-source precision but reveal missing metadata. A reranker may improve hit@1 while adding unacceptable latency. Preserve both outcomes and make the decision from the written acceptance rule.

7. What goes wrong

The prompt is tuned to fix a retrieval miss

Symptom: the expected source is absent from the captured top results, but the team adds "use the correct document" to the answer prompt.

Fix: debug corpus membership, query terms, chunk boundaries, filters, candidate count, and ranking first. A model cannot quote a passage it never received.

A table row is split from its header

Symptom: retrieval returns R-17 | 4 C | 48 h, but neither the model nor reviewer can tell which column is temperature or duration.

Fix: keep the table title and column headers with each row group, preserve page and table identifiers, and add a question whose expected evidence requires the header.

Semantic search is used alone for exact identifiers

Symptom: a query for RD-5 returns conceptually similar reader documents but misses the exact schedule or instrument.

Fix: test lexical plus semantic retrieval, or an exact metadata filter, against identifier cases. Record weights and filters rather than merely naming the mode "hybrid."

Improvements are anecdotes

Symptom: three favorite questions look better after a change, but there is no baseline, denominator, or regression list.

Fix: freeze at least ten representative questions and expected passages, save the before run, and rerun every case after one change.

A citation badge counts as groundedness

Symptom: the cited file is relevant, so a changed number, missing exception, or may-to-must rewrite passes review.

Fix: compare every answer clause with the exact retrieved passage. Score source retrieval and answer support in separate columns.

Several retrieval levers change together

Symptom: chunk size, overlap, embedding model, filters, top k, and reranker all change, producing a higher score with no causal diagnosis.

Fix: write one hypothesis and change one lever. If operational constraints force a bundle, label it as a bundle and do not attribute the result to one component.

8. Do it yourself: a 90-minute retrieval diagnosis

Use one framing, not both. Choose a protocol-like corpus for Lab or a policy-and-contract corpus for Company. Keep it synthetic, public, or explicitly approved and use a non-production index.

Minutes 0-10: state the supported question type, corpus revision, owner, top k, and stop condition. Confirm that you can export source IDs and exact retrieved passages before generation. If you cannot inspect retrieval, stop and add tracing rather than guessing from citations.

Minutes 10-25: write ten questions before changing settings. Include an exact identifier, a heading-dependent fact, a table or list, a qualifier such as may or except, an absent fact, and an ordinary case. For each, name the expected source and passage location. Have the corpus owner confirm those expectations.

Minutes 25-35: define three binary graders: expected passage retrieved, answer supported only by retrieved passages, and answer addresses the question. State how abstentions and missing evidence score. Mark any changed number, lost negation, wrong source version, or changed modal as blocking.

Minutes 35-47: run the baseline with generation disabled and capture the ordered top results. Then run the fixed answer stage over those exact passages. Fill all ten rows and calculate retrieval and groundedness independently. Keep raw failures.

Minutes 47-57: group retrieval misses by cause: source absent, filter exclusion, heading loss, boundary split, exact identifier miss, semantic mismatch, or ranking loss. Choose one group and write one hypothesis. Do not select a fix from a generation failure.

Minutes 57-68: apply exactly one retrieval change: one chunking rule, overlap value, metadata filter, lexical/semantic weight, candidate count, or reranker setting. Record its exact before and after values. Keep the corpus, questions, expected evidence, model, prompt, top k, and graders unchanged.

Minutes 68-80: rerun all ten cases in the same order. Report totals and case-level changes: fixed, regressed, unchanged, and still blocking. Inspect the exact retrieved text for every moved case, not only the aggregate.

Minutes 80-87: ask a colleague or source owner to independently check at least three rows: one pass, one retrieval miss, and one groundedness failure. Resolve an incorrect expected source as a new set revision, not a silent edit to this comparison.

Minutes 87-90: write keep, revert, or investigate for the retrieval change. Name the next owner for any generation or question failure. Remove temporary test indexes and unneeded traces under the approved retention rule.

9. Exit check

Deliver exactly one artifact: one passing retrieval-evaluation report containing ten questions scored on retrieval and groundedness separately, with one retrieval fix applied and re-scored.

It passes when the report identifies the corpus and configuration revisions; preserves all ten questions and expected passages; shows ordered before and after retrieval evidence; records the one changed retrieval setting and everything held fixed; gives separate per-case retrieval, groundedness, and addresses-question results; calculates both run totals; lists fixed, regressed, and remaining blocking cases; includes a human spot-check; and ends with a keep, revert, or investigate decision plus the next owner. The answer stage must remain fixed during this comparison. A retrieval improvement does not pass a release gate while a blocking groundedness failure remains.

The script, test index, and temporary traces are working materials, not additional submitted artifacts. Put the required evidence and decision into the single report.

10. Rule to remember

Find out which half is broken before you fix anything.

11. Further reading & tools

• Taught: T04-L02 - A knowledge base your team can ask - establishes the curated corpus and source register this evaluation assumes.
• Taught: T03-L03 - Test-driven prompting - provides the fixed-set, baseline, grader, and one-change discipline used here.
• Taught: Chat with your own documents - introduces retrieve-then-answer, chunking, grounding, and citation review.
• Taught: AnythingLLM - provides an inspectable workspace exercise for supported and unsupported document questions.
• Taught: Onyx - separates search results, chat answers, document sets, and visibility checks.
• Catalogued: OpenAI Retrieval guide (opens in a new tab) - official documentation for semantic search, source chunks, attribute filtering, ranking options, and score thresholds.
• Catalogued: Weaviate hybrid search (opens in a new tab) - official documentation for combining vector and keyword search and configuring their balance.
• Catalogued: Weaviate reranking (opens in a new tab) - official documentation for reranking an initial result set.
• Catalogued: AnythingLLM document-chat introduction (opens in a new tab) - current official distinction between attaching full text and retrieving workspace chunks; verify controls against the installed version.
• Catalogued: Onyx RAG and Search documentation (opens in a new tab) - official description of search, filters, and cited results in Onyx.
• Catalogued: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (opens in a new tab) - original paper describing the retrieval-plus-generation pattern.
• Catalogued: Tools index - current catalogue entries for AnythingLLM, Onyx, Weaviate, and related retrieval tools.