Keeping a knowledge base alive

2. The source was withdrawn; the answer was not

You are operating the document stack that your lab or company now relies on. In March, the source owner withdrew a protocol after a control failed. Six months later, the assistant still tells a researcher to follow it and cites the old title. In the Company version, procurement withdrew a price list, but a sales colleague receives an answer quoting its obsolete discount.

The source file is gone. The connector dashboard shows a recent successful refresh. Neither fact proves that every indexed chunk, cache, citation, and answer stopped using the document. The system has been adding and updating content, but its deletion path has never been tested. Search history records the question, yet nobody can show which source IDs were returned to which authorised user.

This is Level 5 Operator responsibility: work can stop when stale instructions spread, and the service must survive its original builder leaving. You need a defined withdrawal window, a deletion signal that reaches every searchable copy, a freshness canary that tests known changes, retrieval events that can reconstruct impact without becoming a second document archive, and a runbook another operator can execute.

3. After this you can

• Re-index on a schedule and handle deletions.
• Detect a stale or withdrawn document still being quoted.
• Log what was retrieved and for whom.
• Prove that a withdrawn document stops appearing within a stated window while a current control remains available.

4. Prerequisites

• T04-L04 · Permission-aware retrieval, including two-account negative tests and permission-change timing.
• An approved non-production Onyx connector, Weaviate-backed retrieval service, or equivalent system where you can inspect indexing attempts and ranked source references.
• A synthetic source area and ordinary test account whose documents you may add, replace, and withdraw.
• Rights to trigger the documented refresh and deletion or prune operation without granting the test account administrator access.
• Access to the application retrieval boundary, log destination, scheduler, alert route, and connector stop control.
• A source owner, retrieval operator, privacy or retention owner, and substitute operator.
• About 60 minutes for the independent drill; choose a short test-only service-level objective that fits that time.

Use only the synthetic fixtures below, public material, or explicitly approved data. Do not withdraw a production protocol, price list, contract, manuscript, or record to make the test realistic. Do not place credentials, raw questions, retrieved passages, personal identifiers, customer data, participant data, or restricted titles in the submitted evidence. Test in an isolated collection or connector and bound every query to its synthetic corpus.

5. The idea in one page

A knowledge base is alive only when its searchable state follows the authoritative source in both directions. An update pipeline that only upserts is an archive pretending to be a mirror.

source change -> change detected -> index mutation -> caches invalidated
      |                                  |
      +-> durable source ID + version ---+-> retrieval canary -> alert or pass
                                                        |
                                                        +-> bounded retrieval event

Delta refresh reads changes since a durable checkpoint. It is usually cheaper and faster than rebuilding the corpus, so use it for routine additions and edits. Its contract must include removals. A timestamp-only scan often cannot discover a source that no longer exists; it needs a deletion feed, reconciliation against a complete source-ID inventory, or periodic pruning.

Full re-index rebuilds all eligible objects from the authoritative source. It is useful after a schema, chunker, embedding, or metadata migration and as periodic reconciliation. It costs more compute and can temporarily duplicate, omit, or mix revisions unless you build into a new index and switch only after validation. A full rebuild is not a substitute for a timely withdrawal path.

Tombstoning records that a stable source ID was withdrawn at a particular time and revision. The query path must exclude tombstoned objects immediately, before ranking or generation, while asynchronous workers remove chunks, vectors, previews, and caches. This provides fast denial and an auditable deletion job. Keep the tombstone's minimal identifier and timing only as long as policy requires; do not preserve the withdrawn body inside it.

Define two clocks. The refresh objective says how soon an ordinary source update becomes searchable. The stricter withdrawal objective says how soon a removed source becomes impossible to retrieve or quote. Measure from the authoritative withdrawal event, not from when an operator notices it. Alert if the job does not run, runs partially, loses its checkpoint, or exceeds either objective.

Freshness monitoring needs paired canaries: a withdrawn unique marker that must disappear and a current marker that must remain. Test ranked Search and generated Chat because caches or stored context can differ. Also sample known-changed documents: the current revision must appear and the superseded revision must not.

At the application retrieval boundary, emit a bounded event containing time, request or trace ID, privacy-preserving principal ID, groups or policy revision, corpus and connector revision, query fingerprint or approved canary ID, ordered source IDs and source versions, decision, latency, and retention class. Content is not required to answer “who received which source reference?” Restrict readers, exclude secrets and passages by default, define retention from the investigation need, and verify deletion. Product query history alone may record a question without the ranked source IDs your audit needs.

6. The worked example: withdraw one known document on time

The running Level 5 project is the Level 4 document stack in operation. Keep the permission-aware connector and ordinary account from the previous book, but use a new isolated synthetic corpus. Lab and Company use the same schedule, source IDs, test sequence, logging contract, and pass rule. Only the nouns and marker values change.

Write the operating contract before the clock starts

Create a change record with these fields. Values shown are test objectives, not product defaults or recommendations for every service.

service: synthetic-knowledge-freshness-test
environment: non-production
source_of_truth: "[isolated source location]"
connector_or_collection: "[safe identifier]"
source_owner: "[role]"
retrieval_operator: "[role]"
retention_owner: "[role]"
substitute_operator: "[role]"
delta_refresh: "every 5 minutes in this test"
full_reconciliation: "weekly in the operating design"
withdrawal_objective: "10 minutes from source withdrawal"
surfaces: [search, chat]
principal: "ordinary synthetic account label"
failure_action: "disable this test corpus and notify [route]"
log_retention: "14 days for synthetic drill events"

In a real service, choose frequencies from source volatility, harm, connector capability, corpus size, recovery time, and cost. Record actual observed durations before promising them. The ten-minute objective makes this drill finishable; it does not claim every connector can or should meet ten minutes.

Create a stable-ID source register

Store source identity separately from filename and display title. A rename is an update, not a delete-plus-unrelated-add. Each indexed chunk must carry enough metadata to remove every derivative of one source.

Do not make a filename the only deletion key. Chunking turns one source into many records, and a later filename can differ from the one first indexed. Preserve the stable source ID on every chunk and log it on retrieval.

Lab framing: supersede a protocol safely

Create these two plain-text files in the isolated Lab source:

Source ID: SYN-CURRENT-001
Title: Cedar synthetic protocol status
Version: 3
State: current
Body: The current Lab verification marker is CEDAR-CURRENT-6142.

Source ID: SYN-WITHDRAW-017
Title: Cedar synthetic protocol version one
Version: 1
State: current
Body: The withdrawal marker is CEDAR-WITHDRAW-9285.

This represents the PubMed-and-protocol stack being operated, not an actual scientific instruction. No procedure, sample, participant, or unpublished result is present. The protocol owner remains responsible for real scientific decisions.

Company framing: withdraw a price list safely

Use the parallel fixture in the isolated Company source:

Source ID: SYN-CURRENT-001
Title: Harbor synthetic commercial status
Version: 3
State: current
Body: The current Company verification marker is HARBOR-CURRENT-6142.

Source ID: SYN-WITHDRAW-017
Title: Harbor synthetic price list version one
Version: 1
State: current
Body: The withdrawal marker is HARBOR-WITHDRAW-9285.

This contains no real price, customer, supplier, contract term, or personal data. Commercial owners, not the assistant, decide which real price list is authoritative.

Establish that retrieval works before testing deletion

Run the connector's documented refresh and wait for a successful indexing attempt. Record its attempt ID, connector revision, start and finish times, and checkpoint. “Scheduled” is not completion. A connector in error may leave old indexed content available, so a recent dashboard timestamp alone is not enough.

From the ordinary synthetic account used in T04-L04, query the exact current marker and exact withdrawal marker in Search. Record the ordered stable source IDs returned. Then ask Chat:

Which synthetic source contains [exact marker]? Return its source ID, version, and marker only.

Both markers must be retrievable before withdrawal. This is the positive control. If the future withdrawal query returns nothing but this initial control never succeeded, you have not proved deletion.

At the retrieval boundary, emit events in this shape:

{
  "time": "2026-09-04T10:00:00Z",
  "trace_id": "freshness-lab-001",
  "principal_ref": "sha256:[test-principal-hash]",
  "policy_revision": "synthetic-policy-r4",
  "corpus_revision": "synthetic-corpus-r7",
  "canary_id": "withdraw-marker-before",
  "surface": "search",
  "returned_sources": [
    {"source_id": "SYN-WITHDRAW-017", "source_version": "1", "rank": 1}
  ],
  "outcome": "returned",
  "retention_class": "synthetic-14d"
}

The event deliberately omits the query text and passage. Hashing a principal is useful only when the approved audit service can resolve it during the retention window; an irreversible random label that nobody can map does not support investigation. Keep the mapping separately with narrower access.

Withdraw, tombstone, and reconcile

Record withdrawal_requested_at in UTC. In the source system, remove SYN-WITHDRAW-017 through its approved withdrawal action. Do not merely move it to a folder the connector still scans. Emit or capture a deletion signal carrying the stable source ID and source revision.

If your architecture supports tombstones, write one minimal event:

{
  "source_id": "SYN-WITHDRAW-017",
  "source_version": "1",
  "state": "withdrawn",
  "withdrawn_at": "[UTC time]",
  "source_event_id": "[safe test event ID]"
}

The query filter must exclude this stable source ID immediately. Then let the deletion worker enumerate and remove all matching chunks, vector objects, metadata rows, previews, generated citation caches, and answer caches. In a direct Weaviate design, delete by stable source-ID filter in the correct collection and tenant, inspect a dry run where supported, execute the bounded deletion, and repeat until zero matching objects remain. Account for configured batch limits; one successful request can remove only part of a large source.

Volatile Weaviate behaviour — last verified 2026-09-04: The current primary documentation says deletion requires a collection plus an ID or filter, and also requires a tenant for a multi-tenant collection. Its examples track the latest client and database releases rather than pinning one pair. The same page documents dryRun, verbose output, and a configurable QUERY_MAXIMUM_RESULTS limit whose documented default is 10,000; matches above the effective limit require repeated deletion. It also says collection-less object operations from before Weaviate v1.14 are deprecated and are scheduled for removal in v2.0.0. Treat those versions and defaults as a verification boundary, not a promise about your deployment: record the server and client versions used in the drill, inspect their release notes and API reference, and test the operation in the isolated collection before deletion.

For a managed connector, use the connector's documented removal and prune path rather than deleting vector objects behind its back. Trigger the supported test operation, record its attempt ID, and wait for terminal success. A full re-index is the controlled fallback when the connector cannot prove a complete delta deletion, not the first reaction to every withdrawal.

Volatile Onyx behaviour — last verified 2026-09-04: The current primary administration page distinguishes Refresh Frequency from Prune Frequency, describes pruning as removing old source data that no longer exists, and documents current defaults of 30 minutes and 30 days respectively. The current file-connector OpenAPI page, whose published schema identifies itself as Development rather than a numbered release, documents POST /manage/admin/connector/{connector_id}/files/update as an atomic add/remove operation that triggers indexing and requires manage:connectors or administrator permission. These are documentation-version boundaries, not guarantees for an older self-hosted image, a newer cloud rollout, or every connector type. Before the drill, record the deployed Onyx image or release identifier and connector type, then verify its own administration page or matching OpenAPI schema for removal, prune, permissions, and completion semantics. Never infer that refresh removes missing sources on a deployment where that behaviour has not been verified.

Invalidate caches by stable source ID or corpus revision. Do not depend only on time-to-live: the withdrawal objective may be shorter than the cache lifetime. Do not delete chat history indiscriminately. Instead, prevent old stored context from being reused as current evidence and apply the approved history-retention and deletion rule.

Test disappearance without losing service

Poll safely at a fixed interval, such as once per minute, until the ten-minute objective ends. Each poll has four checks:

1. Search the exact withdrawn marker; SYN-WITHDRAW-017 must not appear in title, snippet, source link, or ranked source IDs.
2. Ask Chat the exact-marker question in a new session; it must not quote, cite, paraphrase, or hint at the withdrawn source.
3. Search the current marker; SYN-CURRENT-001 version 3 must still appear.
4. Inspect the index by stable source ID; zero chunks or objects for SYN-WITHDRAW-017 must remain after physical deletion completes.

The effective stop time is the latest time at which all four conditions first hold, not the worker's “accepted” time. Calculate:

withdrawal duration = all_conditions_met_at - withdrawal_requested_at

If the withdrawn source disappears but the current control also disappears, service is broken rather than fresh. If Search passes while Chat cites a cached answer, the withdrawal has not completed. If the query filter hides the source but physical objects remain, user exposure may be controlled, but the deletion job is still incomplete; record both milestones.

Make freshness observable every day

Schedule the paired canary against an isolated permanent synthetic source. Rotate marker versions through the same approved ingestion process rather than editing the expected result after a failure. Alert on:

• delta refresh not starting by its schedule;
• terminal error, partial completion, or repeated checkpoint;
• current marker absent or wrong version;
• withdrawn marker returned on any surface;
• source-to-index count mismatch beyond the documented tolerance;
• withdrawal objective exceeded;
• retrieval event missing required source, principal, policy, or retention fields.

Run a full reconciliation on the approved rhythm: compare eligible authoritative stable IDs and versions with indexed IDs and versions. Produce missing, extra, wrong_version, and unknown_state sets. Quarantine or investigate extras; do not automatically delete broad matches without a reviewed scope and dry run. Build a new index for major schema changes, validate freshness, permissions, and retrieval quality, then switch the alias or application route atomically and retain the old index only for the approved rollback window.

Complete the single passing-test record

Your one passing-test artifact is withdrawal-drill.json: evidence that a withdrawn document stops appearing in answers within the stated window. It records observations, not copied content:

{
  "service": "synthetic-knowledge-freshness-test",
  "framing": "lab",
  "source_id": "SYN-WITHDRAW-017",
  "source_version": "1",
  "source_revision": "synthetic-source-r8",
  "index_revision": "synthetic-index-r7",
  "principal_ref": "sha256:[test-principal-hash]",
  "current_control": {"source_id": "SYN-CURRENT-001", "version": "3"},
  "withdrawal_objective_seconds": 600,
  "withdrawal_requested_at": "2026-09-04T10:05:00Z",
  "query_denial_at": "2026-09-04T10:09:00Z",
  "physical_deletion_at": "2026-09-04T10:11:00Z",
  "all_conditions_met_at": "2026-09-04T10:11:00Z",
  "duration_seconds": 360,
  "attempt_refs": ["index-test-41", "prune-test-12"],
  "before": {"search_returned_withdrawn": true, "chat_cited_withdrawn": true},
  "after": {
    "search_returned_withdrawn": false,
    "chat_cited_withdrawn": false,
    "withdrawn_objects_remaining": 0,
    "current_control_returned": true
  },
  "retrieval_log_checks": {
    "principal_present": true,
    "ordered_source_ids_present": true,
    "raw_query_or_passage_present": false,
    "retention_class_present": true
  },
  "operator": "Synthetic Retrieval Operator",
  "substitute_retest": "pass",
  "decision": "pass"
}

Replace sample times and outcomes with observations. Never alter a failed result to match this example. Have the substitute operator repeat the after checks using only the runbook, locate the alert and connector stop control, and record the retest in this same artifact. That proves the service can survive the original builder leaving.

7. What goes wrong

Refresh only adds

Symptom: new and edited files appear quickly, but a source missing from the authoritative system remains searchable indefinitely.

Fix: add a deletion feed, stable-ID inventory reconciliation, or supported prune operation. Test a real synthetic withdrawal on Search, Chat, index objects, and caches.

The filename is the identity

Symptom: renaming or moving a document leaves old chunks under one key and adds new chunks under another.

Fix: carry one stable source ID and source version through every chunk, derivative, citation, cache key, deletion signal, and retrieval event.

A full rebuild is the only deletion plan

Symptom: one urgent withdrawal requires hours of recomputation and may mix old and new revisions while users continue querying.

Fix: use immediate query exclusion by tombstone plus bounded physical deletion. Reserve validated full rebuilds for reconciliation and major index changes.

A successful job is mistaken for fresh answers

Symptom: the connector reports success, but Chat still cites cached content or the expected source was outside the connector's actual scope.

Fix: measure from source event to user-visible Search and Chat behaviour, pair the negative marker with a current positive control, and inspect physical remnants separately.

Nobody owns freshness

Symptom: alerts exist, but source, retrieval, and privacy teams each assume another team will respond.

Fix: name source owner, on-call retrieval operator, retention owner, stop authority, and substitute. Rehearse one exceeded-window escalation.

Logs record the question but not retrieval

Symptom: after a stale answer, history shows what the user asked but not the ordered source IDs, versions, corpus revision, or effective principal.

Fix: emit bounded events at the retrieval boundary. Correlate them by trace ID with the answer while excluding raw content and secrets by default.

Audit logs become a shadow corpus

Symptom: full questions, passages, answers, titles, and user identities remain searchable long after their operational purpose ends.

Fix: store references and outcomes, pseudonymise principals with a controlled resolution path, restrict readers, set retention by event class, and verify synthetic-event deletion.

Batch deletion stops early

Symptom: the delete request succeeds, but only the first bounded batch disappears and later chunks remain retrievable.

Fix: inspect matched, deleted, and failed counts; repeat the exact scoped operation until zero objects match; alert on partial or non-decreasing counts.

8. Do it yourself: a 60-minute withdrawal drill

Choose Lab or Company and use one isolated synthetic corpus. Do not test both unless you have separate safe environments.

Minutes 0–8: write the refresh and withdrawal objectives, owners, source of truth, connector or collection, ordinary principal, Search and Chat surfaces, retention class, failure action, and stop control. Confirm the substitute can access the runbook.

Minutes 8–16: create the current and withdrawal fixtures with stable IDs, versions, and unique markers. Run the supported refresh to terminal success. Record the connector revision, checkpoint, attempt ID, and indexed derivative IDs or counts.

Minutes 16–23: retrieve both markers through Search and Chat as the ordinary account. Confirm source IDs and versions. Inspect the correlated retrieval events for principal reference, policy and corpus revisions, ordered source references, outcomes, and absence of raw content.

Minutes 23–28: record the UTC start time and withdraw SYN-WITHDRAW-017 in the authoritative source. Capture its deletion event or tombstone. Trigger only the documented prune, delete, or refresh path.

Minutes 28–40: poll at the written interval. Test withdrawn Search, new-session Chat, current control, and direct stable-ID object count. Record the first query-denial time and physical-deletion time. Stop and execute the failure action if content leaks beyond the objective.

Minutes 40–48: calculate duration and compare it with the prewritten objective. Inspect batch counts, failed objects, cache invalidation, and connector status. Record pass, fail, or escalate without hiding partial completion.

Minutes 48–55: have the substitute operator rerun the four after checks, locate the next scheduled delta and reconciliation jobs, identify the alert recipient, and explain how to stop the test corpus.

Minutes 55–60: finish the single withdrawal-drill.json, remove temporary access, retain or delete synthetic fixtures and events under the written rule, and schedule any unresolved finding with owner and date.

9. Exit check

Deliver exactly one passing-test artifact: withdrawal-drill.json, evidencing that a withdrawn synthetic document stops appearing in answers within a stated window.

It passes when the artifact names the service, framing, source and index revisions, stable withdrawn source ID and version, current control ID and version, ordinary principal reference, prewritten withdrawal objective, source-event time, query-denial time, physical-deletion time, calculated duration, connector or deletion attempt references, before and after Search and Chat results, zero remaining indexed derivatives, current-control success, retrieval-log field checks, operator, substitute retest, and decision. The before control must prove the withdrawn fixture was retrievable. The after checks must show no title, marker, snippet, citation, paraphrase, or source reference from it while the current control remains retrievable.

The artifact fails if the objective was written after the result, only the source file was checked, only one retrieval surface was tested, a connector status substitutes for a query, physical remnants are unexamined, logs cannot associate returned source references with an authorised principal, real sensitive content appears in evidence, or the substitute cannot repeat the check.

10. Rule to remember

Deleting the file does not delete the answer.

11. Further reading & tools

• Taught: T04-L04 · Permission-aware retrieval — establishes the identity, connector, and negative-test boundary this operating drill preserves.
• Taught: Onyx connector administration (opens in a new tab) — primary documentation for refresh frequency, prune frequency, indexing states, failures, and complete re-indexing.
• Taught: Onyx Search API guide (opens in a new tab) — primary documentation for inspecting ranked source results at the retrieval boundary.
• Taught: Weaviate object deletion (opens in a new tab) — primary documentation for deletion by ID or filter, dry runs, verbose results, and bounded batch deletion.
• Catalogued: Neo4j vector indexes (opens in a new tab) — primary documentation for creating, querying, monitoring, and deleting vector indexes in an alternative graph-backed retrieval design.
• Catalogued: Onyx file-connector update API (opens in a new tab) — current atomic add/remove operation for file connectors and its required management permission.
• Catalogued: Onyx prune API (opens in a new tab) — current endpoint for triggering pruning on one connector-credential pair.
• Catalogued: Onyx connector indexing status API (opens in a new tab) — current fields for attempt status, last success, repeated errors, and indexed-document counts.
• Catalogued: Onyx audit logs (opens in a new tab) — primary documentation for structured administrative audit events; pair these with application retrieval events rather than assuming they contain ranked passages.
• Catalogued: Onyx query history (opens in a new tab) — current query-history and export capability; apply an approved access and retention decision.
• Catalogued: T10-L05 · Operating data pipelines — continues with source-to-output provenance and changed-format detection.
• Catalogued: T12-L05 · Governance, evidence and handover — supplies inventory, retention, ownership, and handover controls for the wider service.
• Catalogued: Tools index — compare retrieval platforms only after defining the withdrawal and evidence contract.