Many agents, and memory

2. Three correct-looking answers, one invented fact

The agent from T07-L04 · Agents with real credentials has a narrow identity, read-only inputs, no arbitrary network access, and no authority to publish. It can safely propose a disposition for one synthetic document. Now the queue is growing. You split the job into a retrieval agent, an extraction agent, and a synthesis agent. Each has a tidy prompt and returns fluent output.

In the Lab, the running project reviews synthetic collaborator PDFs. Retrieval selects a fictional study note, extraction turns its claims into fields, and synthesis prepares an evidence summary. In the Company, the same operating design retrieves a synthetic support ticket and policy excerpt, extracts the request and constraints, and drafts a resolution for review. Neither pipeline can file, send, delete, approve, or update a live record.

During one run, the source says that a sample size is not reported. The extractor writes sample_size: 0. The synthesizer treats the field as verified, writes “the study enrolled no participants,” and proposes that memory retain this “fact.” In the Company version, a ticket says that the renewal date is unconfirmed; extraction turns it into a definite date, synthesis confidently promises that date, and the memory candidate would make the mistake recur.

No single worker was asked to make the final unsupported claim. The failure exists in the composition: one hop changed uncertainty, another trusted the changed field, and a memory write threatened to carry it into later runs. Adding more agents has created more boundaries, calls, retries, cost, and places where meaning can change.

Your Level 5 task is not to make the agents debate until one sounds right. It is to operate one bounded system whose partitions, handoffs, memory writes, budgets, and traces make the failing hop identifiable. The passing result is allowed to be incomplete or needs_review. A confident answer without attributable evidence is not success.

3. After this you can

• Split work across agents with clean handoffs.
• Decide what an agent should remember and when to write it.
• Diagnose a failure that only appears with several agents running.
• Decide when serial work is safer or cheaper than a fleet.
• Preserve evidence, uncertainty, authority, and task identity across every hop.
• Operate concurrency, retry, context, retention, and human-review limits without exposing production data or actions.

4. Prerequisites

• T07-L04 · Agents with real credentials, including its preassigned resource, untrusted-content boundary, least-privilege connector, terminal approval state, and sanitized trace.
• T03-L05 · Regression sets and evaluation ops, especially fixed fixtures, intermediate checks, critical-case gates, and replayable traces.
• A local or approved non-production runner that exposes a trace and child spans. A framework is optional.
• Three synthetic Lab documents or three synthetic Company tickets, each with immutable fixture IDs and expected evidence.
• One coordinator identity, three read-or-draft worker roles, a deterministic schema validator, and a named reviewer.
• An approved trace store with access and retention rules. Allow 120 minutes.

Use only public, synthetic, or explicitly approved data. The examples below are fictional. Do not place collaborator manuscripts, participant information, customer tickets, employee records, credentials, private histories, model hidden reasoning, or production traces in the exercise. Use secret references rather than values. Keep all action connectors disabled: the fleet may read supplied fixtures and return drafts, but it may not message, publish, resolve, update, or delete.

Start from the known-good single-agent path. If one bounded agent cannot satisfy the same observable contract on the fixed fixtures, do not split it. Establish that baseline first; otherwise parallelism distributes an unknown defect and makes the comparison meaningless.

5. The idea in one page

Split by independent evidence, not by job titles

Splitting helps when units are independent, expensive enough to overlap, and cheap to merge. Three documents can be extracted independently. One mutable document being edited by three agents cannot. A step whose input depends on the previous answer is serial even if an orchestration product draws it as a fleet.

Estimate the whole path, not model latency alone:

fleet cost = coordinator calls + worker calls + retries + validation + trace storage + human review
elapsed time = serial portions + slowest required parallel branch + merge and review

Parallel work can reduce elapsed time while increasing total usage. Amdahl's law captures the underlying limit: the serial fraction, including aggregation and review, caps speedup. Begin with the smallest useful concurrency, reserve capacity for ordinary service and recovery, and define a stop rule before launch. A timeout is incomplete, not permission to create unbounded replacement workers.

A handoff is a typed claim boundary

Do not pass a conversation summary. Pass a small contract that the receiver validates:

The receiver rejects missing fields, unknown schema versions, changed fixture IDs, unsupported claims, excess authority, and stale deadlines. It does not “helpfully” infer what the sender meant. The coordinator owns the final draft and keeps conflicts visible; a majority vote does not turn three unsupported statements into evidence.

An interoperable protocol can standardize task discovery, messages, artifacts, and state. It does not establish trust, authorization, correctness, or permission to disclose data. Verify endpoint ownership, compatible protocol and schema versions, authenticated identity, local authorization, and failure semantics before a bounded exchange. An advertised Agent Card is capability metadata, not approval.

Memory is first a write decision

Retrieval quality cannot repair a false record admitted earlier. Durable memory needs a write gate before storage:

candidate claim
  -> necessary for named purpose?
  -> supported by inspectable source?
  -> stated fact, inference, or unresolved conflict?
  -> correct subject, access, and valid-time interval?
  -> owner, review date, retention, correction, and deletion path known?
  -> verified memory | quarantined candidate | reject

Never write memory directly from an individual worker's conclusion. The writer receives a validated synthesis artifact plus its source evidence and reviewer state. A useful record includes claim_id, claim, source reference, claim type, confidence, purpose, access class, valid time, recorded time, owner, lifecycle state, and supersession link. valid_time says when the claim applied; recorded_time says when the system learned it. Missing support returns unknown; the nearest old record does not fill a gap.

Durable memory is not runtime context. For each request, select only records whose purpose, access, freshness, confidence, and lifecycle qualify. Exclude the rest before the model call. Retrieved memory remains untrusted content and cannot expand tool authority. Corrections create linked new states and exclude superseded claims from future retrieval; deletion removes active eligibility and follows the separately approved retention policy.

One trace, many attributable spans

Give the root run a trace ID and every agent or deterministic gate a span with a parent. Record fixture references or hashes, contract and schema revisions, agent role, model route or deterministic code revision, input and output artifact references, validation result, state transition, usage, latency, retry number, and error category. Record memory candidates and write decisions as separate spans.

Do not store authorization headers, secret values, complete restricted documents, unrestricted prompts, or hidden model reasoning. A useful trace records observable inputs, outputs, decisions, and evidence—not an invented explanation of internal thought. With this structure, a final wrong statement can be traced backwards to the first span where a supported property changed.

6. The worked example: trace the handoff that erased uncertainty

Build one operating skeleton and apply either skin. The Lab pipeline reviews fictional collaborator evidence; the Company pipeline triages and drafts a fictional resolution. Both use the same three roles, schemas, budgets, and deliberate fault.

Freeze the baseline and partition

Run a known-good single-agent baseline against three fixtures. The observable contract is:

Input: exactly three assigned synthetic items and their approved policy fixture.
Output: one draft review table with a row per item, quoted evidence, and uncertainty.
Actions: read supplied fixtures and draft only.
Critical rule: absent or unconfirmed values remain null and needs_review=true.
Terminal states: completed, incomplete, denied, failed, needs_review.

Require all three critical-rule cases to pass. Save the baseline trace reference, usage, latency, and result in the eventual artifact. Only then introduce the following serial pipeline:

coordinator -> retrieval -> extraction -> synthesis -> memory write gate -> human review

Retrieval is bounded to preassigned fixture IDs. Extraction receives retrieved text and returns source-linked fields. Synthesis receives only validated extraction artifacts and makes a draft table. The memory gate receives only reviewer-eligible candidate claims. These stages are not usefully parallel within one item because each depends on the prior artifact. You may process different fixture IDs concurrently, with at most two items in flight. No worker shares a mutable output; the coordinator alone assembles the draft.

Use these fictional fixtures:

The matching T07-L04 hostile line may remain inside one fixture as untrusted text. Because workers have no send, delete, approval, arbitrary-fetch, or memory-write capability, it cannot add an action. This lesson tests semantic corruption between agents, not by weakening the credential boundary already established.

Define and validate each handoff

Use JSON or equivalent typed records. The extraction handoff should have this shape:

{
  "schema_version": "handoff-1",
  "run_id": "RUN-SYN-005",
  "task_id": "EXTRACT-SYN-01",
  "parent_span_id": "span-retrieve-01",
  "objective": "Extract stated fields from SYN-01 only",
  "input_refs": ["SYN-01@sha256:[recorded-hash]"],
  "verified_claims": [],
  "unknowns": ["target_value not reported"],
  "authority": ["read supplied artifact", "draft extraction"],
  "forbidden": ["browse", "send", "publish", "resolve", "write_memory"],
  "completion_check": "Every non-null value has an exact quote",
  "deadline": "[exercise deadline]"
}

For the Lab, call the nullable field sample_size; for the Company, call it renewal_date. The common validated extraction schema contains:

item_id: immutable fixture ID
target_value: string, number, or null
value_status: stated | uncertain | absent | conflicting | withdrawn
evidence_quote: exact supplied passage or null
source_ref: immutable fixture reference
needs_review: boolean

The deterministic validator applies these rules before synthesis:

1. item_id and source_ref must match the assigned input.
2. A non-null value requires an exact evidence substring in that source.
3. absent, uncertain, conflicting, or withdrawn requires needs_review: true.
4. absent or withdrawn cannot carry a current value.
5. Additional keys, unknown states, missing trace identity, or forbidden actions reject the handoff.

The synthesis agent cannot repair or waive a rejected extraction. The coordinator marks that item incomplete and routes the source and rejection reason to review.

Run once without the fault

Set concurrency to two items, one attempt per stage, a fixed per-worker response budget, and a wall-time limit appropriate to the local test runner. Record limits before starting. Launch SYN-01 and SYN-02; launch SYN-03 only when one slot is free. The expected SYN-01 extraction is null and review-required. SYN-02 preserves its stated value. SYN-03 is withdrawn and review-required.

The synthesis output is a draft, not a decision. Each row retains task ID, source, evidence, extraction status, and any conflict. The coordinator cannot convert missing work into “no issue.” The human reviewer can reopen every source from its approved fixture reference.

Compare fleet and single-agent results on exactly the same fixtures. Record total worker calls, coordinator calls, usage units, wall time, validation work, and review rows. If the fleet provides no material elapsed-time or quality benefit, the correct operating decision may be to keep the single agent. Architectural complexity is not an exit requirement.

Inject one deliberate failure

Now enable one deterministic test-only fault in the extraction adapter for SYN-01:

if test_fault == "erase-uncertainty" and item_id == "SYN-01":
    target_value = 0                 # Lab
    # or target_value = "2026-10-01" # Company
    value_status = "stated"
    needs_review = false

This fault never calls a model, network, credential, or production service. It deliberately changes the typed artifact after extraction and before validation. Keep the original source unchanged. Run the same fixtures, revisions, limits, and expectations; change only the named fault flag.

The validator must reject SYN-01 because the non-null value has no exact supporting quote. Synthesis receives no accepted extraction for that item and returns incomplete, not a confident sentence. The memory gate receives no candidate. A passing run therefore contains a deliberate internal failure and a safe final state.

Isolate the first bad hop

Read the trace from root to leaves rather than inspecting only the final draft:

The extraction adapter is the failing component. Retrieval preserved the evidence; validation contained the error; synthesis and memory behaved correctly. “The multi-agent system hallucinated” is too broad to operate. The trace identifies the exact contract boundary and revision to repair.

Add the fault case to the regression set from T03-L05. Repair the extraction path—for example, remove the faulty transformation and enforce nullable fields before serialization—then rerun with the fault disabled. Do not delete the failing trace or relax the quote check. The corrected run must again return null plus review for SYN-01.

Admit, retrieve, correct, and delete memory

Only a validated, human-accepted claim may become a write candidate. Use SYN-02, never the failed SYN-01, to exercise the path:

claim_id: MEM-SYN-02-v1
claim: [stated synthetic value]
source_ref: SYN-02@sha256:[recorded-hash]
claim_type: stated-in-source
confidence: confirmed-for-exercise
purpose: draft this review only
access: synthetic-review-team
valid_from: 2026-09-04
valid_to: 2026-09-30
recorded_at: [observed run time]
owner: exercise-review-owner
status: active
review_or_delete_on: 2026-09-30

Test four decisions. First, the supported candidate is admitted after named review. Second, the invented SYN-01 candidate is rejected and remains absent from every active index. Third, a request with the wrong purpose or access class excludes MEM-SYN-02-v1 before model context is assembled. Fourth, create a synthetic correction MEM-SYN-02-v2, link supersedes: MEM-SYN-02-v1, and mark v1 ineligible for future retrieval without silently rewriting its evidence. Finally, run the approved synthetic deletion or expiry path and verify that the record is absent from active retrieval; record any separately authorised audit retention honestly.

Tools such as Letta or Mem0 can implement memory operations, and CrewAI can orchestrate roles, but their presence proves none of these decisions. Configure the same external gates, identities, schemas, lifecycle states, and traces around whichever implementation you use. Use Langfuse or an approved equivalent to correlate spans; verify field capture and redaction rather than assuming an integration captures what operators need.

Bound retries and recovery

Retry only a failure known to be safe and plausibly transient. A schema rejection is deterministic for the same artifact and revision; do not retry it with more agents. A timeout before any side effect may receive one bounded retry with a new span linked to the same task and attempt number. Because this exercise exposes no side-effecting tools, duplicate external action is impossible. In a future system, use idempotency keys and reconcile unknown outcomes before retrying.

Stop the run when the call, usage, time, validation-error, or concurrency budget is reached. Cancel outstanding workers through the runner's supported control, mark their tasks incomplete, and retain terminal evidence. Do not fall back to a broader model route, credential, context window, or untraced worker.

7. What goes wrong

The fleet appears before the baseline works

Symptom: three agents disagree, but nobody knows whether the original contract or fixture was sound.

Fix: run one bounded agent against the fixed set first. Split only a passing observable task, and compare the fleet with that baseline.

A fluent summary replaces the handoff

Symptom: the receiver gets “the source confirms zero” without a source ID, quote, uncertainty state, schema revision, or authority boundary.

Fix: reject untyped handoffs. Separate verified claims, assumptions, and unknowns; require evidence and immutable references for every value.

Parallel outputs lose ownership and become votes

Symptom: agents overwrite the same review table or ticket, or the coordinator accepts an unsupported date merely because two workers repeat it.

Fix: give each worker immutable input and a separate output artifact. Let one coordinator aggregate evidence—not votes—into a draft, preserve conflicts, and keep live changes behind human approval.

Memory is written from an unverified conclusion

Symptom: a worker's convenient guess appears in later runs as established context.

Fix: remove direct memory authority from workers. Require source-backed validation, purpose and access checks, lifecycle fields, and named review at the write gate.

Old memory quietly becomes current truth

Symptom: retrieval returns a superseded status or uses a value outside its valid interval.

Fix: filter by lifecycle and valid time before context construction. Link corrections, preserve provenance, and return unknown when no interval qualifies.

Every retry multiplies cost

Symptom: timeouts launch replacement workers until the provider or review queue is saturated.

Fix: set attempt, concurrency, usage, and wall-time budgets before launch. Retry only classified transient failures; stop deterministic failures and unknown side-effect outcomes.

The trace shows an answer but not its path

Symptom: the final span is red, while retrieval, handoff transformations, validation, and memory decisions are invisible.

Fix: correlate one root trace with parented spans and artifact references at every boundary. Capture state changes, revisions, usage, and validation—not secrets or hidden reasoning.

Debugging creates a second sensitive archive

Symptom: complete documents, tickets, credentials, and long-lived memory values are copied into traces.

Fix: use fixture IDs, hashes, bounded evidence snippets, structured statuses, redaction tests, access controls, and expiry. Confirm deletion with a synthetic trace before using approved data.

8. Do it yourself: a 120-minute multi-agent failure drill

Maintain one trace-and-isolation packet throughout; the steps below are sections of that artifact, not separate deliverables.

Minutes 0–15: choose Lab or Company. Record the owner, reviewer, synthetic fixtures, baseline contract, allowed and forbidden actions, stop control, trace access, and retention. Confirm that no production connector or real record is reachable.

Minutes 15–30: run the single-agent baseline on SYN-01 through SYN-03. Preserve exact revisions, expected and observed results, usage, latency, and a trace reference. Stop if the critical uncertainty and correction cases do not pass.

Minutes 30–48: define retrieval, extraction, synthesis, coordinator, and memory-gate responsibilities. Write the handoff and extraction schemas, validators, terminal states, per-worker budget, concurrency of two, and retry rule. Prove workers cannot send, publish, resolve, delete, or write memory.

Minutes 48–65: run the fault-free fleet. Verify parent-child trace correlation, immutable fixture references, exact evidence, visible unknowns, safe aggregation, and zero direct memory writes. Compare elapsed time and total usage with the baseline.

Minutes 65–82: enable only erase-uncertainty for SYN-01 and rerun. Preserve the unchanged source, faulty extraction artifact, validator rejection, incomplete synthesis state, and absent memory candidate. Do not improve the fixture or prompt after seeing the result.

Minutes 82–95: identify the first divergent span. Record expected property, observed property, component and revision, downstream containment, impact, and owner. Repair the extraction path, disable the fault, and rerun the same case to its expected null-and-review state.

Minutes 95–108: use only supported SYN-02 to test memory admission, purpose/access exclusion, synthetic correction and supersession, expiry or deletion from active retrieval, and any authorised retention exception. Search active memory for the invented SYN-01 value and require no match.

Minutes 108–116: reconcile root and child span counts, total calls, attempts, usage, latency, and terminal states. Run a synthetic canary search to confirm that secrets and prohibited full content are absent. Record the trace retention and deletion owner.

Minutes 116–120: have a second operator use only the packet to point to the first bad hop, explain why later stages stopped safely, locate the regression case, and identify the fleet stop control. Remove temporary access and expire test state.

If safe tracing, cancellation, or synthetic isolation is unavailable, stop and document the unmet prerequisite. Do not compensate with production data, broad credentials, uncontrolled load, or fabricated observations.

9. Exit check

Deliver exactly one artifact: one multi-agent trace-and-isolation packet for either the Lab review pipeline or Company triage-and-resolve pipeline.

It passes when the packet contains the fixed three-fixture contract; passing single-agent baseline; role and partition map; versioned handoff schemas; authority, concurrency, call, usage, retry, and stop limits; one fault-free multi-agent trace; one trace with the deliberate erase-uncertainty failure; and one corrected rerun. The root trace and parented spans must show the source remained correct, the extraction hop first invented a value, deterministic validation rejected it, synthesis ended safely, and no memory write occurred. The packet must also show one supported memory admission, one purpose or access exclusion, one correction or supersession, one verified removal from active retrieval, a cost and latency comparison, sanitized evidence, and the second operator's review.

It fails if the problem is attributed only to “the model,” the first divergent hop cannot be named, a timeout or rejected handoff becomes success, workers share mutable state or consequential authority, unsupported agreement wins by vote, the invented value reaches active memory, retries exceed the written budget, traces contain secrets or unapproved data, or the evidence requires an oral explanation from the original builder.

A safe incomplete final answer passes when the trace proves that the deliberate internal failure was isolated and contained. Repair and rerun within the same packet; do not create a second artifact to hide the failed evidence.

10. Rule to remember

Two agents can be confidently wrong in a way one could not.

11. Further reading & tools

• Taught: T07-L04 · Agents with real credentials — supplies the least-privilege, untrusted-content, approval, sandbox, and trace boundary retained by this fleet.
• Taught: T03-L05 · Regression sets and evaluation ops — turns the deliberate cross-agent failure into a fixed intermediate regression case.
• Taught: Handoffs — defines evidence-linked transfer of objective, verified state, open work, authority, and completion checks.
• Taught: Agent interop — separates protocol-compatible tasks and artifacts from identity, authorization, and trust.
• Taught: Several AI workers on one job — covers partitioning, concurrency budgets, provenance, and evidence-based aggregation.
• Taught: Heidelberg Marvin memory — provides the illustrative write gate, retrieval gate, correction, and deletion lifecycle applied to synthetic records here.
• Taught: Memory is a write problem — provides the admission, provenance, correction, and deletion decisions used here.
• Taught: Temporal knowledge graphs — distinguishes valid time from recorded time and preserves corrections without rewriting history.
• Taught: Runtime context — filters durable records by purpose, access, freshness, and uncertainty before a model request.
• Catalogued: A2A Protocol specification (opens in a new tab) — primary specification for agent discovery, task lifecycle, messages, artifacts, bindings, and security considerations.
• Catalogued: Langfuse observability and application tracing (opens in a new tab) — primary product guidance for correlated traces, observations, metadata, and usage capture.
• Catalogued: CrewAI documentation (opens in a new tab) — primary product documentation for crews, flows, tasks, guardrails, and observability integrations.
• Catalogued: Letta memory documentation (opens in a new tab) — primary product guidance for agent memory blocks and memory operations.
• Catalogued: Mem0 documentation (opens in a new tab) — primary product guidance for memory extraction, storage, search, update, and deletion APIs.
• Catalogued: W3C OWL-Time (opens in a new tab) and W3C PROV-O (opens in a new tab) — primary standards for temporal concepts and provenance representation.
• Catalogued: NIST AI Risk Management Framework (opens in a new tab) — primary framework for mapping, measuring, managing, and governing operational AI risk.