▸How-to Scientist +1
When I need an answer backed by the exact paper and passage
A literature assistant that answers from your shelf — and shows its sources — instead of inventing plausible-sounding references.
Track 9 · Intermediate
9.1
Chat with your own documents
Make AI answer from your own files — with citations
1Overview
Retrieval-Augmented Generation (RAG): instead of relying on the model's memory, give it your own documents to answer from. Taught as a weak → strong pipeline — chunk & embed, retrieve & rank (hybrid search + reranking), then ground every answer in sources with citations and measure retrieval quality. With a vector-store cheat sheet, persona examples, and primary-sourced FAQs and videos.
In this chapter you will learn how to build AI assistants that answer questions strictly from your own documents, always providing verifiable citations. You will be able to create chatbots for literature review, customer support, policy guidance, financial reporting, and internal procedures that retrieve exact passages, verify sources, and flag missing evidence, while measuring and reducing hallucinations.
1.1After this chapter you can
→Chunk and embed documents so the right passage is actually findable
→Retrieve with hybrid search and rerank down to the few chunks that matter
→Ground every answer in sources with citations — and catch it when it doesn't
→Measure retrieval quality (precision, recall, faithfulness) and diagnose failures
1.2What is Retrieval‑Augmented Generation?
It’s a pipeline that feeds the model your own documents—by chunking, embedding, retrieving, ranking, and then grounding each answer in those sources with citations.
1.3How are relevant passages found?
You split files into chunks, embed them, run a hybrid search to retrieve candidates, and optionally rerank them for higher relevance before using them to answer.
1.4Why add citations to answers?
Citations let you verify that each response is directly supported by the retrieved documents, letting you measure retrieval quality and maintain trust in the output.
1.5The moves — weak → strong ladder
- 1Chunk on structure, not a fixed character count
- 2Small chunks to search, parent doc to answer
- 3Add context to each chunk before embedding (contextual retrieval)
- 4Retrieve top-k from a vector store, don't paste everything
- 5Hybrid search: keyword + vector
- 6Rerank candidates before the LLM sees them
- 7Order results to beat "lost in the middle"
- 8Force grounded answers with citations
- 9Measure retrieval, not just the answer
- 10Score faithfulness to catch hallucination
- 11Diagnose the standard failure modes
2Techniques
Learn
Chunk & embed
Turn documents into findable pieces
Chunk on structure, not a fixed character count How you split a document decides what can ever be retrieved. Slice on its natural boundaries, not an arbitrary character budget that cuts mid-thought.
✕Instead of
Split every 1,000 characters, cutting sentences and tables mid-thought.
✓ Try this 💬 AI chat
Split on the document's own structure — headings, paragraphs, sections — and let chunks overlap by ~10–20% so a thought is never orphaned across a boundary.
Why it works: Bad chunking dooms retrieval no matter how good the embeddings are: if the answer is split across two chunks, neither one looks relevant. Splitting on structure with ~10–20% overlap keeps each chunk a coherent, self-contained unit.
Small chunks to search, parent doc to answer The best size for finding a passage is not the best size for answering from it. Decouple them.
✕Instead of
Use one chunk size for both retrieval and the answer — big enough to answer means imprecise search; small enough to search means too little context.
✓ Try this 💬 AI chat
Embed small, precise chunks (~100–400 tokens) for retrieval, but when one hits, return its parent section so the model has enough context to actually answer.
Why it works: Small chunks make retrieval precise (the matching passage stands out); returning the parent section gives the model the surrounding context it needs to write a grounded answer. You get precise retrieval AND enough context.
Add context to each chunk before embedding (contextual retrieval) A chunk ripped out of its document loses what it was about. Tell each chunk what it is before you index it.
✕Instead of
Embed each chunk in isolation: "it grew 3% this quarter" — what grew? which quarter? which company?
✓ Try this 💬 AI chat
Before indexing, prepend a short LLM-written explanation that situates the chunk in its document (e.g. "This is from Acme's Q2 2026 report; revenue grew 3% vs Q1"), then embed that.
Why it works: Anthropic measured that this "contextual embeddings" step cuts the rate of top-20 retrieval failures by 35% — a large gain from a cheap preprocessing pass, because each chunk now carries the context that makes it findable.
Retrieve & rank
Find the few chunks that matter
Retrieve top-k from a vector store, don't paste everything The point of RAG is to bring only the relevant passages into the window — not to dump the whole corpus and hope.
✕Instead of
Paste all your documents into the prompt every time and let the model sift.
✓ Try this 💬 AI chat
Embed every chunk once into a vector store, then at query time embed the question and retrieve only the top-k most similar chunks.
Why it works: Retrieving the top-k most similar passages means only the relevant material reaches the context window — cheaper, faster, and less distracting for the model than a corpus dump that buries the answer in noise.
Hybrid search: keyword + vector Semantic search understands meaning but fumbles exact tokens. Run both kinds of search and fuse them.
✕Instead of
Use semantic (vector) search only — it misses exact codes, IDs, error strings, and product names that share no "meaning" with the query.
✓ Try this 💬 AI chat
Run vector search and keyword search (BM25) together, then fuse the two rankings into one list.
Why it works: Dense (vector) retrieval catches meaning; sparse (keyword/BM25) retrieval catches exact tokens like SKUs and error codes. Anthropic's contextual embeddings + contextual BM25 together cut top-20 retrieval failures by 49% — further than embeddings alone.
Rerank candidates before the LLM sees them A fast first pass gets you a rough shortlist; a slower, sharper model picks the real winners from it.
✕Instead of
Feed the top-k straight from the vector store into the model and hope the order is right.
✓ Try this 💬 AI chat
Over-retrieve a wide candidate set (top 20–100), pass it through a reranker (a cross-encoder that scores each chunk against the query), and keep only the best 5–10 for the prompt.
Why it works: Anthropic measured that adding a reranking step cut retrieval failures by 67% — their best result. Over-retrieve cheaply, then let an accurate reranker pick the handful that actually go to the model.
Order results to beat "lost in the middle" Where a chunk sits in the prompt changes how well the model uses it. Put your best evidence where the model looks hardest.
✕Instead of
Place the most relevant chunk in the middle of a long list of retrieved passages.
✓ Try this 💬 AI chat
Keep the retrieved list short, and place the strongest chunks at the very start and very end of it.
Why it works: Liu et al. ("Lost in the Middle", arXiv 2307.03172) showed models attend best to the beginning and end of their context and degrade when key information sits in the middle — true even for long-context models. A short list, best chunks at the edges, sidesteps the dip.
Ground & evaluate
Answer from sources, then prove it
Force grounded answers with citations An answer is only trustworthy if you can trace each claim to a source. Make the model cite — and make a missing citation visible.
✕Instead of
Prompt "answer the question" over the retrieved text and trust whatever comes back.
✓ Try this 💬 AI chat
Instruct the model to answer ONLY from the provided sources, cite the specific chunk for each claim, and say "not in the sources" when the answer isn't there.
Why it works: When every claim must carry a citation, a missing or wrong one becomes a visible failure you can catch — instead of a silent hallucination that reads just as confidently as a grounded answer.
Measure retrieval, not just the answer If the right chunk never got retrieved, no amount of prompt-tuning saves the answer. Score the retriever directly.
✕Instead of
Eyeball a few final answers and decide the system "seems fine".
✓ Try this 💬 AI chat
On a labelled query set, score context precision (were the retrieved chunks relevant, and ranked high?) and context recall (did you retrieve everything needed to answer?).
Why it works: The answer cannot be right if retrieval failed — so you measure the retriever on its own. Low precision means noisy context; low recall means the evidence never arrived. Either one caps how good the answer can be.
Score faithfulness to catch hallucination A cited-looking answer can still invent claims the sources never made. Check, claim by claim, that the answer is supported.
✕Instead of
Assume any answer that includes citations is actually grounded in them.
✓ Try this 💬 AI chat
Compute faithfulness = (claims supported by the retrieved context) / (total claims), using an LLM judge to check each claim against the sources.
Why it works: Faithfulness (a standard Ragas metric) catches the confident claims the retrieved sources never actually supported — the hallucinations that slip past a glance at the citations. (The Evals chapter teaches the general judge-validation loop this builds on.)
Diagnose the standard failure modes RAG breaks in specific, nameable places. Triage which stage failed instead of reaching for a bigger model.
✕Instead of
Conclude "it's wrong, let's use a bigger model" whenever an answer is bad.
✓ Try this 💬 AI chat
Triage the stage: a stale index, bad chunking, a retrieval miss (low precision/recall), lost-in-the-middle ordering, or ungrounded generation (low faithfulness) — and fix the stage that actually failed.
Why it works: RAG failures are stage-specific, and naming the stage tells you what to fix. A bigger model does nothing for a stale index or a retrieval miss; the cure for each failure mode is different.
3Lessons 7
3.1 Upload a PDF to Adobe Acrobat AI chat and get a cited answer
Adobe Acrobat’s online AI‑powered PDF chat tool that lets you ask questions about uploaded documents.
You will be able to upload a PDF, ask a question, and receive an answer with numbered source links that highlight the original text.
- Open the Adobe Acrobat AI chat page and click the Select files button.
- Drag and drop a PDF (or DOCX, PPTX, TXT, RTF) into the upload zone.
- In the chat box, type a question such as “What are the most important sections?” and press Enter.
- When the answer appears, click one of the numbered reference links to see the highlighted source passage in the PDF.
- You'll see An AI‑generated response that includes numbered citations; clicking a number highlights the exact sentence or paragraph in the uploaded document.
- Takeaway Citations let you verify AI answers instantly by linking each claim back to its original location in the source file.
3.2 Build a searchable AI knowledge base from PDF files
A vector store that holds overlapping sections from your PDF documents.
Create an indexed vector store where each chunk is linked to its source file and heading
- Collect the PDFs you need and place them in a single folder
- Split each PDF into overlapping sections using the document’s headings
- Generate embeddings for every section with an embedding API
- Insert the embeddings together with metadata (file name, heading, page range) into the vector store
- You'll see The vector store lists entries that can be queried, each showing the original PDF name and its section heading
- Takeaway Chunking on logical document boundaries preserves traceability, enabling precise citations later
- Check What information does each entry in the created vector store contain that enables later citation of PDF passages?
3.3 Generate answers that cite exact document sections
A retrieval‑augmented generation pipeline that selects the most relevant chunks before prompting the model.
Produce a response that either cites the relevant PDF passages or clearly states that no answer is found in the documents
- Send the user query to the vector store and retrieve the top 50 candidate chunks (use hybrid search if available)
- Run a reranker model over those candidates and keep the five highest‑scoring chunks
- Prompt the language model with the original query and the selected chunks, instructing it to include each chunk’s source metadata in its answer
- If none of the retained chunks contain an answer, have the model output a clear “not in our docs” message
- You'll see The reply shows hyperlinks or citation strings pointing to the specific file and heading, or displays a “not in our docs” notice
- Takeaway Retrieving many chunks then reranking narrows the context to truly relevant evidence reducing hallucinations
- Check How does applying a reranker to the top retrieved chunks help reduce hallucinations in the final answer?
3.4 Ingest a PDF into Azure OpenAI On Your Data and run a semantic query
Azure OpenAI On Your Data, a service that chunks, embeds, and indexes uploaded files for retrieval‑augmented generation.
You will upload a PDF to Azure’s portal, let it be indexed, and retrieve relevant text using a natural‑language prompt.
- Sign in to the Microsoft Foundry (classic) portal and navigate to Azure OpenAI On Your Data.
- Use the Upload files option to select the same PDF you used with Adobe and confirm the upload.
- Wait for the service to finish ingesting, chunking, and embedding the document into an Azure AI Search index.
- In the web‑based chat interface, type a query like “Summarize the key points of this document” and submit it.
- You'll see A response that includes excerpts pulled from the indexed PDF, with references to the source chunks used for generation.
- Takeaway Embedding your documents in a vector store enables semantic search and grounding of LLM answers without fine‑tuning.
3.5 Compare citation formats between Adobe AI chat and Azure OpenAI responses
A side‑by‑side evaluation of how two platforms surface source citations for the same document.
You will identify differences in citation style, clickability, and traceability between Adobe’s numbered links and Azure’s chunk references.
- Return to the Adobe chat window and note the format of its numbered citations (e.g., [1], [2]).
- In the Azure OpenAI response, locate the reference identifiers that point to specific indexed chunks.
- Click a citation in each platform to verify that the highlighted text matches the answer claim.
- Record the observed differences: Adobe shows inline clickable numbers that jump to PDF highlights; Azure lists chunk IDs with excerpt snippets.
- You'll see A clear list of how each system presents source links and whether clicking them reveals the original passage.
- Takeaway Understanding citation mechanics helps you choose the right tool for workflows that require transparent provenance.
3.6 Create a citation‑based quality report of answer faithfulness
An evaluation routine that compares each claim in an answer against the retrieved source chunks.
Produce a report that quantifies how many statements are backed by retrieved evidence
- Collect a set of test questions and run them through the pipeline built in Lesson 2, storing both the answers and the cited chunks
- Split each answer into individual factual claims (sentences or bullet points)
- Verify whether each claim appears verbatim or is paraphrased from any of the cited chunks, using a human judge or an automated script that checks string overlap
- Calculate the proportion of supported claims and output a summary report showing overall faithfulness and examples of unsupported statements
- You'll see A concise report listing the percentage of grounded claims per question and highlighting any hallucinated sentences
- Takeaway Systematic faithfulness scoring lets you identify retrieval gaps and iteratively improve chunking, ranking or prompting strategies
- Check What metric is calculated to quantify how many statements in an answer are backed by the retrieved source chunks?
3.7 Create a simple citation‑based quality report for an AI answer
A short markdown document that evaluates answer faithfulness by checking each cited passage against the source.
You will produce a report that lists every citation, shows the extracted source text, and marks whether the AI’s claim is fully supported.
- Open a plain‑text editor and start a new markdown file named
quality_report.md. - Copy the AI answer from Adobe (or Azure) into the file under a heading Answer.
- For each numbered citation, paste the highlighted source paragraph you saw when clicking the link, labeling it Source #n.
- Add a bullet‑point assessment after each source: “Supported”, “Partially supported”, or “Not supported” based on whether the claim matches the text.
- Save the file and preview it to ensure all citations and assessments are visible.
- You'll see A markdown report that pairs every AI claim with its exact source excerpt and a clear faithfulness label.
- Takeaway Documenting citation checks creates an audit trail that reduces hallucinations and builds trust in RAG‑based assistants.
4You’ll know it worked 34 checkable outcomes in this chapter
- ✓The assistant cites the correct PDF and passage when answering a question
- ✓The draft contains only numbers that appear in the metrics docs and each number is cited
- ✓Bot returns the correct answer to previously failing questions
- ✓User receives quoted SOP steps with source citation in response to a query
- ✓When asked about an unlisted expense, the bot replies "not in policy"
- ✓Reps receive an answer that cites the exact product sheet or battlecard used
- ✓A table appears listing each contract, clause number, and exact quoted text for change-of-control, assignment, termination or drag-along clauses, each row citing its source page
- ✓HR dashboard shows salary range per role with linked placement records
34 outcomes in all — one per recipe below.
5FAQ, Tips & How-to 45
one problem, one solution, one action
Research & data tools3
▸How-to Scientist +1
A measured hallucination rate for your research bot, so you fix grounding before a colleague trusts a made-up finding.
▸How-to Scientist
Every candidate annotation arrives with the marker list and the source behind it, so you argue with the evidence instead of with a model’s recollection of what CD14 means.
Knowledge & docs9
▸How-to Scientist +1
Need a summary with every sentence backed by a source
A summary where every line is traceable to a real source — no confident sentences floating free of the evidence.
▸How-to Founder +2
Need an investor update with only real numbers
Draft updates where every figure is pulled from your real numbers — no rounded-up guesses sneaking into investor comms.
▸How-to Small biz +1
Need exact policy answers
Staff get correct, citable answers about leave, expenses and conduct without pinging you — and the bot won't invent a policy.
▸How-to Support +1
Citations point to unseen sources
Citations you can trust, because every one is verified against what was actually retrieved.
▸How-to Finance +1
Consistent, citable policy answers for the team — with honest gaps instead of confident improvisation.
▸How-to HR / People +1
Employees ask about benefits
Employees get fast, correct, sourced answers about benefits and policy without an HR ticket each time.
▸How-to HR / People +1
New hire asks onboarding questions
New starters get reliable, sourced onboarding answers and a clear "ask a human" signal for the gaps.
▸How-to Investor +1
Need exact answers for diligence
You can interrogate hundreds of documents in natural language and trace every answer back to its exact source before it goes into your IC memo.
▸How-to Physician
A fast, citable starting point for a guideline or formulary lookup — every answer traces to the primary source, which the clinician checks before it informs a decision. Not a diagnostic tool.
Customer & client portals3
▸How-to Founder +1
Customers want source for every reply
A support chatbot customers (and you) can trust, because every answer points back to a real article.
▸How-to Support +1
Repetitive support tickets need answers
A self-serve bot that resolves the easy, repeated questions correctly and with a source — freeing agents for the hard ones.
▸How-to Support +1
Getting lots of noisy support articles
Sharper answers from the same knowledge base, because the model only sees the chunks that actually matter.
Internal tools & ops8
▸How-to Founder +1
You avoid building a RAG pipeline you don't need yet — and know exactly when you will.
▸How-to Small biz +1
Fast, sourced answers about your own contracts — without scrolling 40-page PDFs or guessing at terms.
▸How-to Small biz +1
Bot can’t find obvious info in docs
Retrieval that actually surfaces the right passage, fixed at the chunking stage rather than by swapping models.
▸How-to Operations +1
Need exact steps from our runbooks
Tribal knowledge becomes a searchable assistant that answers from the actual SOP, not from someone's memory.
▸How-to Operations +1
When SOPs change, answers stay outdated
Answers always reflect the current document, not last quarter's — no silent drift as procedures change.
▸How-to HR / People +1
HR bot blends unrelated policies into replies
Cleaner context means clearer answers; you stop the bot from blending three unrelated policies into one wrong reply.
▸How-to Investor +1
Want all change‑of‑control, assignment, termination and drag‑along clauses with sources
A structured clause matrix in minutes instead of days of manual review, with every entry traceable to the original contract page.
▸How-to Physician +1
Can’t find the right SOP step
Fast, sourced answers to routine protocol questions, freeing staff from hunting through binders — with every answer traceable to the actual SOP, not a memory of it.
Dashboards & analytics5
▸How-to Operations +1
You see which topics the retriever silently misses, instead of blaming the model for answers it never had the context to give.
▸How-to Finance +2
Sourced answers about specific numbers in long filings — with the exact table cited, not a paraphrase.
▸How-to Finance +1
Need variance commentary that only uses figures
Variance notes where each figure traces to the actual report — no transcription errors or invented deltas.
▸How-to Investor +1
Hard to spot emerging themes across holdings
A searchable view across your whole portfolio so emerging themes surface before they become surprises, with every signal traced to its source report.
▸How-to HR / People +1
Defensible salary ranges backed by your own data, not a number the model guessed from training — every figure cites the comparable placements it came from.
CRM & sales5
▸How-to Sales +1
Reps need quick, on‑message answers from product docs
Reps get fast, on-message answers grounded in approved collateral — not a hallucinated feature claim in front of a prospect.
▸How-to Sales +1
Need an RFP fast but only using vetted answers
Faster RFPs assembled from vetted answers, with a citation trail back to the approved source for each one.
▸How-to Sales +1
Need accurate account briefing without guesswork
Reps walk into calls briefed from real account history, with each fact traceable to a note — no made-up context about the customer.
▸How-to HR / People +1
Finding exact‑match candidates in a massive CV pool
You surface relevant candidates in seconds from a corpus too large to browse manually, with each suggestion backed by the actual profile text.
▸How-to HR / People
Need a draft candidate shortlist that matches my job description
A ranked draft shortlist grounded in actual profile text, ready for the recruiter to review and refine — not a black-box score.
Content & marketing1
▸How-to Creator +1
Want to lift bits from my old posts
New drafts that sound like you and reuse your real prior facts, with each borrowed point traceable to the original post.
▸How-to Everyone
When a text chunk loses its document context
A chunk ripped out of its document loses what it was about. Tell each chunk what it is before you index it. Anthropic measured that this "contextual embeddings" step cuts the rate of top-20 retrieval failures by 35% — a large gain from a cheap preprocessing pass, because each chunk now carries the context that makes it findable.
▸How-to Everyone
Precise search gives tiny hit but no context
The best size for finding a passage is not the best size for answering from it. Decouple them. Small chunks make retrieval precise (the matching passage stands out); returning the parent section gives the model the surrounding context it needs to write a grounded answer. You get precise retrieval AND enough context.
▸How-to Everyone
Document splits cut sentences in half
How you split a document decides what can ever be retrieved. Slice on its natural boundaries, not an arbitrary character budget that cuts mid-thought. Bad chunking dooms retrieval no matter how good the embeddings are: if the answer is split across two chunks, neither one looks relevant. Splitting on structure with ~10–20% overlap keeps each chunk a coherent, self-contained unit.
▸How-to Everyone
Can't tell if answers are made up
An answer is only trustworthy if you can trace each claim to a source. Make the model cite — and make a missing citation visible. When every claim must carry a citation, a missing or wrong one becomes a visible failure you can catch — instead of a silent hallucination that reads just as confidently as a grounded answer.
▸How-to Everyone
RAG breaks in specific, nameable places. Triage which stage failed instead of reaching for a bigger model. RAG failures are stage-specific, and naming the stage tells you what to fix. A bigger model does nothing for a stale index or a retrieval miss; the cure for each failure mode is different.
▸How-to Everyone
Answers look cited but still make up facts
A cited-looking answer can still invent claims the sources never made. Check, claim by claim, that the answer is supported. Faithfulness (a standard Ragas metric) catches the confident claims the retrieved sources never actually supported — the hallucinations that slip past a glance at the citations. (The Evals chapter teaches the general judge-validation loop this builds on.)
▸How-to Everyone
Can't see why answers are wrong
If the right chunk never got retrieved, no amount of prompt-tuning saves the answer. Score the retriever directly. The answer cannot be right if retrieval failed — so you measure the retriever on its own. Low precision means noisy context; low recall means the evidence never arrived. Either one caps how good the answer can be.
▸How-to Everyone
IDs or error codes get missed by meaning‑only search
Semantic search understands meaning but fumbles exact tokens. Run both kinds of search and fuse them. Dense (vector) retrieval catches meaning; sparse (keyword/BM25) retrieval catches exact tokens like SKUs and error codes. Anthropic's contextual embeddings + contextual BM25 together cut top-20 retrieval failures by 49% — further than embeddings alone.
▸How-to Everyone
Key evidence gets stuck in the middle of a long list
Where a chunk sits in the prompt changes how well the model uses it. Put your best evidence where the model looks hardest. Liu et al. ("Lost in the Middle", arXiv 2307.03172) showed models attend best to the beginning and end of their context and degrade when key information sits in the middle — true even for long-context models. A short list, best chunks at the edges, sidesteps the dip.
▸How-to Everyone
A fast first pass gets you a rough shortlist; a slower, sharper model picks the real winners from it. Anthropic measured that adding a reranking step cut retrieval failures by 67% — their best result. Over-retrieve cheaply, then let an accurate reranker pick the handful that actually go to the model.
▸How-to Everyone
Dumping the whole corpus into the prompt
The point of RAG is to bring only the relevant passages into the window — not to dump the whole corpus and hope. Retrieving the top-k most similar passages means only the relevant material reaches the context window — cheaper, faster, and less distracting for the model than a corpus dump that buries the answer in noise.
The same set on /recipes, filtered by tool and role.
6Videos 3
What is Retrieval-Augmented Generation (RAG)?
A clear, vendor-neutral explainer of what RAG is and why it grounds answers in your own data. Start here for the mental model.
Learn RAG From Scratch – Python AI Tutorial from a LangChain Engineer
A hands-on, code-along build of a full RAG pipeline in Python, taught by a LangChain engineer — chunking, embedding, retrieval, generation.
Reliable, fully local RAG agents with LLaMA3.2-3b
Building a reliable, fully local RAG agent — routing, grading retrieved docs, and checking groundedness — the evaluation-minded end of the lab.
7FAQ 8
What is RAG, and why not just use a bigger context window?
RAG (Retrieval-Augmented Generation) retrieves the few relevant passages from your own documents and puts them in the prompt, so the model answers from your sources instead of its training memory. Anthropic's own guidance: for a knowledge base under ~200k tokens, you can just include all of it in the prompt — no RAG needed. Above that, retrieval is what keeps you from blowing past the window (and paying for tokens the answer never needs).
When does "just use the big context window" beat RAG in 2026?
When your whole knowledge base comfortably fits the window. Anthropic's rule of thumb is roughly: under ~200k tokens (about 500 pages), just put the documents in the prompt — it is simpler and avoids a retrieval pipeline entirely. RAG earns its keep once the corpus is too big to fit, changes often, or is large enough that paying to send all of it on every call is wasteful.
How should I chunk my documents?
Split on the document's own structure (headings, paragraphs, sections) rather than a fixed character count, and let chunks overlap by ~10–20% so a thought is never cut across a boundary. A strong pattern is to embed small, precise chunks for retrieval but return the larger parent section to the model so it has enough context to answer. Good chunking is upstream of everything — bad chunks cap how good retrieval can ever be.
What is hybrid search, and why add keyword search to vectors?
Hybrid search runs semantic (vector) retrieval and keyword (BM25) retrieval together and fuses the results. Vectors catch meaning but miss exact tokens — product codes, IDs, error strings, names — while keyword search nails those exact matches. Anthropic found that contextual embeddings combined with contextual BM25 cut top-20 retrieval failures by 49%, more than embeddings alone.
What is reranking, and is it worth the extra step?
Reranking over-retrieves a wide candidate set (say top 20–100), then passes it through a more accurate model — a cross-encoder reranker that scores each chunk against the query — and keeps only the best 5–10 for the prompt. Anthropic measured that adding reranking cut retrieval failures by 67%, the best result in their study. The first pass is cheap and wide; the reranker is the precision step.
Why does my RAG miss facts even when they're in the documents?
A common cause is "lost in the middle": Liu et al. showed that language models use information best when it appears at the start or end of the input, and degrade when the relevant passage sits in the middle of a long context — even for long-context models. Keep the retrieved list short and put your strongest chunks at the beginning and end. (Other causes: a stale index, bad chunking, or a retrieval miss — triage the stage.)
How do I evaluate a RAG system — what are context precision/recall and faithfulness?
Measure the retriever and the answer separately. Context precision asks whether the retrieved chunks were relevant and ranked high; context recall asks whether you retrieved everything needed to answer. Faithfulness scores the generation: it is the fraction of the answer's claims that are actually supported by the retrieved context, so it catches confident statements the sources never made. These are standard Ragas metrics on a labelled query set.
Which vector store should I use?
If you already run Postgres, start with pgvector — it adds vector similarity search to the database you already have, with no new service to operate. Reach for a dedicated store when scale or features justify it: Chroma for the simplest local prototyping, Qdrant for cheap self-hosted scale, Pinecone for a fully managed (paid) option, or Weaviate for open-source with built-in vectorization and hybrid search.
8Glossary 12 terms
Show the 12 terms
Pipeline
Chunk overlap- Letting adjacent chunks share ~10–20% of their text so a sentence or idea is never cut cleanly across a boundary and lost to retrieval.
Contextual retrieval- Prepending a short, document-aware explanation to each chunk before embedding it, so the chunk carries the context that makes it findable.
Retrieval
Top-k- The k most similar chunks returned for a query — the small set you actually put in the prompt instead of the whole corpus.
Dense vs sparse retrieval- Dense = vector/embedding search that matches meaning; sparse = keyword search (e.g. BM25) that matches exact tokens. Hybrid uses both.
BM25- A classic keyword-ranking algorithm that scores documents by term overlap — strong at exact matches like codes, IDs and names that vectors miss.
Hybrid search- Running vector (dense) and keyword (sparse/BM25) search together and fusing the rankings, so you catch both meaning and exact tokens.
Reciprocal Rank Fusion (RRF)- A simple way to merge two ranked lists (e.g. vector and BM25) into one by combining each item's reciprocal rank in each list.
Reranker / reranking- A second, more accurate pass that re-scores an over-retrieved candidate set against the query and keeps only the best few chunks.
Cross-encoder- A model that reads the query and a candidate chunk together to score their relevance — slower but more accurate than embedding similarity, used for reranking.
Evaluation
Context precision- Of the chunks you retrieved, how many were actually relevant (and ranked high)? Low precision means noisy, distracting context.
Context recall- Of the information needed to answer, how much did you actually retrieve? Low recall means the evidence never reached the model.
Faithfulness- The fraction of an answer's claims that are supported by the retrieved context — catches confident statements the sources never made.
9See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.