Heidelberg AICurriculum
Track 15 · Advanced
15.1

Heidelberg Marvin memory

A memory system for AI sessions, taken apart piece by piece

9 lessons 2026-08-08 AI-generated

1Overview

A memory layer bolted onto AI coding sessions: everything said is captured, indexed by meaning as well as by keyword, and the most relevant passages are injected back into the next prompt automatically.

A full walkthrough of one working memory system — the one running behind the Heidelberg team's own AI sessions. Three tiers of memory and the rule for which tier a fact goes in; capture hooks that must never slow a prompt down; embeddings, and what "matched by meaning" actually computes; a hybrid index with reserved slots so short-term chatter cannot crowd out reviewed lessons; the recall@5 gate that decides whether any of it is working. → The chapter ends on the part most write-ups skip: the promotion loop that turns a conversation into a durable, reviewed lesson, which is the only thing that makes a memory compound instead of expire. → Tool-agnostic: the pieces are a search index, an embedding endpoint and two hooks, and every one of them has free alternatives.

1.2After this chapter you can
→Explain why a context window is not memory, and what a memory layer has to add
→Route a fact to the right tier: reviewed lesson, session chatter, or open ticket
→Capture from a live session without ever delaying or breaking a prompt
→Describe what an embedding is and why query and document text are embedded differently
→Fuse keyword and semantic retrieval, and reserve slots so one tier cannot crowd out another
→Treat the embedding model as a measured choice, not a version to upgrade
→Hold a retriever to a recall@5 floor that fails loudly instead of degrading quietly
→Close the loop: distil expiring conversations into candidate lessons a human approves in seconds
1.3When to reach for it

When you keep re-explaining the same project to the same assistant, and when the thing you needed to remember was written down but nobody could find it again.

1.4Key parts

A capture hook per prompt, a hybrid search index, an embedding endpoint, reserved result slots per memory tier, an expiry job, a recall@5 eval with a failing floor, and a promotion step with a human in it.

1.5What it costs

Nothing per token if the embedding model runs locally. The real cost is context: the recalled passages sit in every prompt for the rest of the session, so the budget is bytes multiplied by remaining turns.

1.6The part that decides everything

Promotion. Capture, recall and expiry are the easy half — a system without a review step that a human will actually use is a short-lived cache wearing a memory's clothes.

Marvin — what writes, what reads, what deletes, and what compounds Capture, hybrid index, recall into the prompt, and the promotion loop that turns an expiring conversation into a reviewed lesson. Marvin — a memory that outlives the session what writes · what reads · what deletes — and the one edge that makes it compound CAPTURE — every session Prompt hook your message + its project Answer hook the reply, in chunks fire-and-forget — a memory write never delays a prompt Embedding endpoint text → 768 numbers, local stored chunks and each question, tagged apart The index hybrid: keyword + meaning blended half and half reviewed lessons grown by promotion chatter + open tickets hundreds of lines a day RECALL — every prompt Top 5 passages reserved slots per tier each passage capped tier + date shown, always Into your prompt before the model reads it recall@5 gate — the build fails below its floor, so a dead retriever stays visible LIFECYCLE — what expires, and what is kept before it does Expire — past a weekend, not before verified, backed up, deleted Distill candidates a local model proposes lessons worth keeping Human review approve → durable lesson one click, and revertible Capture and recall are the easy half. Promotion is the edge that compounds: without it you have a short-lived cache; with it, a memory. expiring chatter

2Lessons 9

2.1 Distinguish between a context window and persistent memory

A persistent storage file holds captured notes beyond the temporary context window, allowing later retrieval through meaning‑based metadata.

Separate the fleeting context window from a lasting store that captures and recalls information by meaning

  1. Identify the context window as the temporary workspace that resets when a session ends
  2. Create a persistent storage file to capture information you want to retain beyond the session
  3. Add metadata or semantic tags to the stored notes so they can be recalled by meaning later
  4. Run the same query twice: first against a fresh assistant, then after loading the persisted memory layer
  • You'll see The same question answered once with no prior knowledge and once after the system has remembered your project
  • Takeaway A context window is cleared each session, while memory requires both capture of events and meaning‑based retrieval
  • Check How does creating a persistent storage file with semantic tags enable the system to recall information by meaning after a session ends?
  • Cost Free to reason about. Building it costs a search index and an embedding endpoint, both of which have open-source, run-it-yourself options.

2.2 Route facts to the correct memory tier

The canonical tier stores reviewed lesson files, ensuring they outrank raw session transcripts and open‑ticket references during retrieval.

Place each fact in its proper tier so that reviewed information always outranks unreviewed data

  1. Assign reviewed lesson files to the canonical tier
  2. Label automatically captured session transcripts with the episodic tier
  3. Index open issue references under the live tickets tier
  • You'll see Recalled passages appear with a tier label and date, preventing session text from being mistaken for an approved rule
  • Takeaway Separate reviewed lessons, raw session logs, and open work into distinct tiers to avoid volume‑driven dominance
  • Check What effect does assigning reviewed lesson files to the canonical tier have on the ranking of recalled passages compared to unreviewed data?
  • Cost A naming convention and one decision per fact. The expensive version is skipping it and letting the unreviewed tier grow until nobody trusts any tier.

2.3 Capture sessions without slowing prompts

A fire‑and‑forget hook writes prompts or reply chunks to memory instantly without waiting for acknowledgement, preserving real‑time interaction.

Write every prompt and answer to memory instantly while keeping the conversation flowing

  1. Configure the prompt‑submission hook to write the user message to memory using fire‑and‑forget semantics
  2. Configure the assistant‑completion hook to split the reply into small chunks and write each chunk to memory as fire‑and‑forget
  3. Add the project field to every memory entry so lines are scoped to their respective project
  • You'll see A prompt is sent and a new document appears in the index before the answer starts streaming, with no measurable delay
  • Takeaway Memory writes must be fire‑and‑forget and tagged with their project so conversations stay distinct
  • Check How do fire‑and‑forget hooks for prompt submission and assistant completion keep conversation flow uninterrupted while writing each exchange to memory?
  • Cost One webhook call per prompt and one per answer. Budget a couple of seconds of timeout that you expect never to spend.

2.4 Compare meanings using embeddings

Tagging distinguishes search queries from search documents, guiding the embedding process to treat them appropriately for cosine similarity scoring.

Generate vector representations of text and measure their similarity to see how meaning is matched.

  1. Generate an embedding for a piece of text, producing a 768‑dimensional vector.
  2. Calculate the cosine similarity between two embeddings to obtain a single similarity score.
  3. Tag query text as a search query and document text as a search document before embedding them.
  • You'll see Two unrelated sentences receive a high similarity score while a keyword search finds no matches.
  • Takeaway Embeddings place meaning in a coordinate space and cosine measures closeness; tag queries and documents differently and cache document vectors by content hash
  • Check Why must query text be tagged as a search query and document text as a search document before generating embeddings for similarity comparison?
  • Cost Free if the embedding model runs on your own machine. Re-embedding an entire small corpus is a job measured in seconds, not hours.

2.5 Create a balanced retrieval index

Reserved slots allocate fixed positions in the result list for each memory tier, guaranteeing balanced representation of keyword and semantic matches.

Build a search system that combines keyword and semantic signals without any tier dominating the results

  1. Enable the hybrid search mode to score both keyword matches and vector matches
  2. Assign output positions using reserved slots so each tier receives its own place in the top list
  3. Limit each injected passage with capped size before storing it in context
  4. Activate per‑session dedupe for lesson sections while leaving session chatter untouched
  • You'll see Results show a label indicating whether they came from fused, keyword‑only or unavailable sources
  • Takeaway Fuse keyword and semantic, reserve a slot per tier, cap every passage, and never let a dedupe rule silently empty your results
  • Check How does reserving slots for each tier in hybrid search prevent any single tier from dominating the top‑ranked results?
  • Cost One small search service. Two days of rolling capture plus a curated corpus is a few thousand documents, which is a non-event for any modern index.

2.6 Evaluate which embedding model works best for your data

A re‑embedding pass rewrites every document with the selected model in one operation, maintaining consistent vector dimensionality for the index.

Identify the better embedding model for your corpus and re‑embed all documents when you switch

  1. Run a head‑to‑head evaluation on your real corpus, measuring top‑five recall for each question
  2. Compare the total scores to see which model performs better overall
  3. Re‑embed every document with the chosen model in a single pass before updating the vector index
  4. Verify that language‑specific queries return relevant results by testing with multilingual files
  • You'll see A two‑column recall table where the newer model wins one question, loses two, and loses the total
  • Takeaway Measure an embedding model on your own corpus before swapping it, and re‑embed everything in the same pass — equal dimensions hide an incompatible space
  • Check What steps are required to reliably switch to a better embedding model across an entire corpus without breaking existing vector dimensions?
  • Cost An afternoon to build the question-to-file fixture. Re-embedding a small corpus to test the alternative is seconds of compute.

2.7 Validate retriever performance and enforce a recall floor

The recall@5 evaluation script measures top‑five retrieval accuracy and triggers a non‑zero exit status if the result falls beneath the preset minimum.

Run a recall@5 evaluation, expose any failure modes, and stop the build when performance drops below a defined threshold

  1. EXECUTE the recall@5 script against the live endpoint, supplying your question‑file pairs
  2. VERIFY that the gate exits non-zero below its floor by checking the process exit status after the run
  3. INSPECT the list of missed queries to distinguish genuine retrieval errors from overly strict fixture mappings
  4. ADD a secondary fixture for coverage checks if you need to detect missing facts in the corpus
  5. CONFIGURE the fallback to be honest by ensuring timeout paths label results as "keyword only" instead of reporting success
  • You'll see The evaluation prints a recall@5 score and aborts the build if the score falls under the floor
  • Takeaway Measure retrieval with recall@5 and treat an unlabelled fallback as a critical failure
  • Check How does the recall@5 script enforce a performance floor by exiting non‑zero when the score drops below the defined threshold?
  • Cost An afternoon for the fixture, seconds per run. Cheaper than one incident caused by memory that was quietly off.

2.8 Add a falsifiable status line to every answer

A falsifiable status line is a single header added to every answer, containing separate claims that visibly change when any memory component fails.

Show a live memory marker at the start of each reply so failures are visible

  1. Insert a one‑line payload at the start of each session containing four independent claims
  2. Configure the assistant to echo that exact line verbatim as the first characters of every reply
  3. Design each claim so it can fail independently, producing a visibly different marker when something is broken
  • You'll see Each answer begins with a line like “🌻🦉 15:13 - marvin 2039 -> live 2340 +365 | recall PASS 93%, 2h ago” and any deviation signals a problem
  • Takeaway A visible marker makes silent memory failures impossible to miss
  • Check What purpose does inserting a one‑line payload with four independent claims at the start of each reply serve for monitoring memory health?
  • Cost One line of context per turn, and a few lines in the hook that already runs. Cheaper than one afternoon spent trusting an assistant that had no memory loaded.

2.9 Promote an expiring conversation into a reviewed lesson

The promotion workflow runs a nightly distillation job, generates a ticket with candidate lessons, and upon approval commits the reviewed file to the version‑controlled tier.

Create a promotion step that turns conversations about to expire into durable lessons

  1. Run the nightly distillation job that scans conversations nearing expiry
  2. Open the generated ticket containing candidate lessons
  3. Click the Approve checkbox for a single candidate lesson
  4. Confirm the promotion by pressing the Commit button, which writes the reviewed file to the version‑controlled tier
  5. Allow the hourly indexer to pick up the new lesson so it can be recalled in future sessions
  • You'll see A candidate lesson appears after the nightly job, you approve it with one click, and it is later recalled by meaning in a different session
  • Takeaway Memory compounds only when an expiring conversation is reviewed and promoted, otherwise the loop stays open
  • Check Which actions transform an expiring conversation into a reviewed lesson that becomes searchable by meaning in future sessions?
  • Cost A nightly run of a local model over expiring conversations, and a review surface. No paid tokens. The expensive part is not compute — it is making approval fast enough that it actually happens.

3See also

Chat with your own documents— the same retrieval machinery, pointed at your files instead of your sessions —Context Engineering— what context does for a model, which is what memory delivers automatically —Building complex codebases— the hand-written version of the same idea: a knowledge layer checked into the repo —

💬 Discuss this chapter

Ask, share, or report — over on the Heidelberg AI community forum.

Ask a question Stuck on something? The community can help. Show what you built Share it — others can try it and cheer it on. Report a problem or idea Spotted a bug, or have an improvement?