Keep a Connected Knowledge Base Current, and Prove a Withdrawn Document Is Gone

T04-L05 · Your documents & knowledge · Level 5 Operator · 18-minute read, plus 4 hours of doing

The file was gone, the answer wasn't

In March a group leader withdrew version 3 of the standard digest SOP, the standard operating procedure for that lab step, after a quality control failed, and put version 4 in its place. The file service shows only version 4. Six months later a new master's student asks the department's assistant how long to digest, and gets the old answer, with SOP-12 v3 as the citation. The nightly sync had added version 4 and never removed version 3. It had reported success every night.

In a company the purchasing team withdraws a supplier price list after the supplier raised prices, and puts the new one on the SharePoint site. Two weeks later a sales colleague asks Microsoft Copilot for the discount on a product line and gets last year's figure. The old file is gone from SharePoint. But the old list is still attached to forty emails in people's mailboxes, and Copilot reads mailboxes too.

You are at Level 5 Operator. Four groups or a whole sales team rely on the assistant every day, and a withdrawn document that keeps being quoted spreads mistakes quietly, for months. This book shows you how to keep the assistant in step with its sources: what to watch, what to do when something withdrawn is still quoted, who takes over, and what you can show an auditor.

After this you can

• Set a refresh window and a stricter withdrawal window for each connected source, and make the sync remove what the source removed.
• Watch four signals every week: the sync ran, the counts match, a withdrawn marker is gone while a current one is found, and no document is past its review date.
• Handle a withdrawal incident: take the document out of the assistant within the window, find who received it, and tell them.
• Keep a withdrawal log and retrieval records an auditor can check, without turning them into a second copy of everyone's questions.
• Hand the operation to a substitute who has run a withdrawal drill from the runbook without you.

Before you start

• Books: T01-L01 for the four-quadrant method. T04-L04 for the connected assistant this book keeps running: its collections, groups, test accounts and marker documents. T04-L03 for the test questions that check answers are right.
• The connected assistant from T04-L04. The lab example uses Open WebUI on the department's server; the company example uses Microsoft Copilot.
• The people involved: the data owner of each source, IT, your data protection officer, and the elected staff representatives: in the lab, the institute's staff council; in a company, the works council.
• A substitute: a named colleague who will run this when you're away or gone, with three hours in their calendar over the next month.
• About 4 hours 20 minutes: 18 to read, about 4 hours to do, over a month.

Last verified 23 September 2026. Open WebUI's official sync tool, oikb, removes documents deleted from the source. Microsoft Copilot's audit records list the files it used for each answer. Details for experts at the end of this book.

The idea in one page

A connected knowledge base is alive only if it follows its sources in both directions: what's added appears, and what's removed disappears. Most syncs are built and tested for the first direction only.

Two clocks, written down. The refresh window says how soon a new or changed document can be asked; a day is usually fine. The withdrawal window says how soon a removed document must stop appearing in answers; it's stricter, because a withdrawn SOP or price list does harm. Measure both from the moment the source changed, not from when you noticed.

Copies live in more places than the collection. A withdrawn document can still reach answers through the collection the sync fills, through files people attached to their own chats, through a custom assistant with the text pasted into its instructions, and, in Microsoft 365, through emails and chats that carry the old file as an attachment. The sync can only clean the first. For the rest you need a rule, and a check.

Four signals, weekly.

The canary pair is the heart of it. It's two small marker documents, like the ones in T04-L04: one current, which must always be found, and one withdrawn in the source, which must never be found. The current one proves the check works; only then does the missing withdrawn one mean removed.

Record references, not content. When something withdrawn was quoted, you'll need to know who received it. Record which documents were used for whom and when, not the full questions and answers. Agree with the data protection officer and the staff or works council who may look, for what reason and for how long.

Everything you do, someone else can do. A runbook, a short page of steps per situation, and a named substitute who has run a withdrawal drill from it while you watched.

Worked example: withdraw one document on time

Both versions follow the four quadrants from T01-L01. Follow the one closer to your work. The people, markers and numbers are examples written for this book.

Lab: the department's SOP assistant

Structure it (15 min). Mira, the technician who reran Jonas's checks in T04-L03, now operates the department's document assistant with IT, since Jonas moved to a new project. Four groups use it; about 40 people ask it every week. After the SOP-12 incident she writes: "A withdrawn SOP must stop being quoted within one working day, anywhere in the assistant. When one was quoted, we must know who got it and tell them. A good result: a weekly check shows it, and Paul from IT can run the whole thing from the runbook."

Understand it (60 min). She starts with the sync. IT's script copies new and changed files from each group folder into its knowledge base every night. It never removes anything. She compares counts: SOPs Cedar holds 41 documents, the Cedar folder 34. Seven withdrawn SOPs are still answering questions.

Then she looks for copies outside the collections. Administrators see every saved custom assistant under Workspace, then Models, and reading their instructions turns up one, Digest helper, with the full text of SOP-12 v3 pasted into its instructions, shared with the Cedar group. And people have attached SOP files directly to their own chats; those copies stay in the chats that hold them.

Last, she asks who received SOP-12 v3 since March. Open WebUI stores each chat with its citations in its database, so IT could search every chat there for the title. But that means reading colleagues' chats. The data protection officer and the staff council agree to one procedure: a search for a withdrawn document's title, run by two named people, only after a withdrawal, with the result used only to tell the people affected. The search finds 11 people.

This changes the problem. She'd thought of it as "fix the sync". The sync is one of three routes, and the other two, pasted text and attached files, no sync can reach. Back to Structure it: "The sync removes what the source removed. Custom assistants link to collections, never paste SOP text. Every SOP carries its version and review date in its first line, so an old copy shows its age in any answer. And a weekly canary pair proves the first part."

Choose tools (15 min).

• oikb, Open WebUI's official sync tool, replaces IT's script. It compares each folder with its knowledge base and removes files deleted from the folder, as well as adding new ones. IT runs it every hour for each group, with the same read-only accounts as before, so the refresh and withdrawal windows are both about an hour. It acts as the Open WebUI user whose key it holds, and can change only the knowledge bases that user may write to.
• The weekly check is a short script IT schedules on the server. It records whether each hourly sync finished, compares counts, and lists SOPs past their review date. For the canary pair it asks Open WebUI's API, its interface for programs, with the key of the test account from T04-L04, so it sees exactly what that ordinary account sees. It posts the result to the department's chat channel every Monday.
• Open WebUI's audit log, set with one server setting (named in Details for experts) to record who called what and when, but not the content of questions. The chat search agreed with the staff council covers the rest.
• The alternative, a search product with built-in retrieval logs such as Onyx, would record which documents were retrieved for whom automatically. For four groups on a server IT already runs, the procedure is enough.

Build it: the sync and the clean-up (60 min). IT installs oikb on the server with pip install oikb, using Python's installer.

oikb needs two things per group. One is the knowledge base's ID, the long code at the end of its address in the browser. The other is an API key, a secret that lets a program act as one Open WebUI user, for the sync's own user, created under Settings, then Account, then API keys.

The first run per group is a dry run, which lists what it would change without changing anything:

oikb sync /data/groups/cedar --kb-id <Cedar's ID> --dry-run

Mira checks the list for Cedar with the group leader: seven withdrawn SOPs, nothing else. Then the same command without --dry-run, and a scheduled job that repeats it every hour. Counts match in all four collections.

The Digest helper assistant's pasted text is replaced by a link to SOPs Cedar. A line goes into the department's assistant rules: SOP text is never pasted into an assistant's instructions. The four group leaders add a first line to every SOP: SOP-12 · version 4 · valid from 12 March 2026 · review by March 2027.

The 11 people who received the old digest time get an email from the Cedar group leader with the correct value and the date the SOP changed. Two had used it; one experiment is repeated.

Build it: the canary pair and the weekly check (45 min). Each knowledge base gets two marker files in its source folder:

canary-current.txt   marker CANARY-CURRENT-CEDAR-3310  (never removed)
canary-withdrawn.txt marker CANARY-GONE-CEDAR-7702     (removed on the 1st of every month)

On the first of each month the check script re-adds the withdrawn file, waits for a sync, confirms it can be found, removes it from the folder, and records how long until the test account stops finding it. Every Monday it confirms again that the current marker is found and the withdrawn one isn't. A Monday post looks like this:

SOP assistant, week 41
Syncs: 168 of 168 finished     Counts: 4 of 4 collections match
Canary: current found in 4/4, withdrawn found in 0/4
Last withdrawal drill: 1 Oct, gone after 52 minutes (window: 1 working day)
Past review date: SOP-07 Orbit (Feb 2026) -> Orbit group leader

The last line goes to the data owner, not to Mira: whether an overdue SOP is still right is the group's decision.

Build it: the withdrawal runbook (30 min). Four sections, each a few lines. A routine withdrawal: the group removes the file from its folder; the next sync removes it; the Monday post confirms. An urgent withdrawal, when a wrong SOP could harm people or samples: an IT administrator opens the knowledge base under Workspace, then Knowledge, deletes the file from it at once with the file's menu, then removes it from the folder. It was quoted: the agreed chat search, the email from the group leader, a line in the withdrawal log. The check failed: what each line of the Monday post means and who to call.

The withdrawal log is a table in the runbook's folder: document, version, who withdrew it and when, when the assistant stopped showing it, who received it, and when they were told. It's what an auditor will ask for.

Build it: handover. Paul from IT runs the October drill from the runbook while Mira watches without helping. He re-adds the withdrawn canary, confirms it's found, removes it, and times it: 48 minutes. Then he does an urgent withdrawal on a test SOP by hand. He gets stuck once: the runbook doesn't say where the test account's password is kept. It's in IT's password manager, and the runbook now says so.

In November the institute's quality officer asks the auditor's question: "How do you make sure staff don't work from withdrawn SOPs when they use the assistant?" Mira hands over the withdrawal log, the last eight Monday posts and the October drill record. The answer takes ten minutes.

What it taught her: the sync was an afternoon's fix. The copies outside it, pasted and attached, needed a rule people would follow, and a first line that shows an old SOP's age.

Company: price lists in Microsoft Copilot

Structure it. Selin, who took over from Markus in T04-L04, operates Microsoft Copilot for the 30-person sales and service team. "A withdrawn price list must stop being quoted within one working day. When it was quoted, we know to whom and correct it. My deputy Tim can run all of it."

Understand it. She checks where the old list lives. The SharePoint file is gone. But Copilot uses what each person can open, and that includes their own mailbox: 43 emails carry the old list as an attachment. Nobody can reasonably delete attachments from 30 mailboxes. She also finds that the price list's filename has never included a date.

To find who received the old figure, IT searches the audit log in Microsoft Purview, Microsoft 365's compliance portal, where every Copilot interaction is recorded with the files it used to answer. In the Purview portal IT opens Audit, searches for the activity CopilotInteraction over the two weeks, exports the results and filters them for the old file's name. The records list the file and the person, not the full conversation. Six people got answers based on the old list in two weeks; two sent quotes with the old discount. The works council agreed to the Copilot pilot on the condition that usage is reported by team, so the data protection officer and the works council agree on the same kind of procedure as in the lab: a search by file, after a withdrawal, used only to correct.

Back to Structure it: "The source removes the old list; that part works. The copies in mailboxes can't be removed, so every price list must say in its first line and its filename when it expires, and people send links, not attachments."

Choose tools. Microsoft Copilot stays. The Purview audit log answers who received it. Price lists get a filename and first line such as Price list Supplier A · valid 1 Oct 2026 to 30 Sep 2027, and the purchasing team shares them as links to the SharePoint file. A current and a withdrawn canary price list, with made-up products, go on the sales site.

Build it. The two quotes are corrected and the customers contacted the same day. Every Monday Selin asks Copilot for both canary products through a test account; on the first of the month she withdraws the canary and times it. The first drill takes three hours from removal to disappearance, within a working day. The withdrawal log follows the lab's pattern. Tim runs the November drill from the runbook alone. When the company's ISO 9001 auditor, who checks its quality management against that standard, asks how staff are kept from using obsolete documents, Selin shows the log, the drill records and the filename rule.

In both cases the sync or the platform did its part once someone checked it. What kept old documents alive was copies nobody had counted. A marker that should have disappeared, checked every week, is what shows the rest is working.

What goes wrong

The sync only adds. Symptom: new documents appear quickly, withdrawn ones never leave, and the sync reports success. Fix: use a sync that removes deletions, or compare counts after every run. Test it with a withdrawn canary every month.

Success is taken from the job's status. Symptom: the sync says "finished", yet answers still cite the old version. Fix: measure from the source change to what an ordinary account sees in an answer, with a current marker in the same check.

Copies outside the collection. Symptom: the document is gone from the source and the collection, and still quoted. Fix: find pasted text in custom assistants, attachments in chats and mail. Link instead of paste. Put the version and expiry in every document's first line and filename.

Nobody knows who received it. Symptom: a withdrawn document was quoted for months, and nobody can say to whom. Fix: keep records of which documents were used for whom, and an agreed procedure for searching them after a withdrawal.

The records become a second archive. Symptom: every question and answer from every colleague is kept forever, readable by all of IT. Fix: record references, not content; two named people may search, for a stated reason; delete on a schedule someone checks.

Freshness has no owner. Symptom: the Monday post lists overdue SOPs for months, and nobody acts. Fix: every line of the check goes to a named person. Overdue documents go to their data owner, not to the operator.

The drill never happens. Symptom: the first real urgent withdrawal is also the first time anyone tries it, on a Friday evening. Fix: a monthly canary withdrawal, and an urgent withdrawal practised once by the substitute.

Do it yourself (4 hours, over a month)

1. Structure it (15 min). For each connected source, write the refresh window, the withdrawal window, and who takes over when you're away.
2. Understand it (60 min). Compare document counts between each source and its collection. Look for copies outside the collections: pasted text, attachments, mail. Find out whether you could say who received a withdrawn document, and agree the procedure with your data protection officer and staff or works council.
3. Choose tools (15 min). Pick a sync that removes deletions, decide where the weekly check runs and where it posts, and decide where records of use are kept and for how long.
4. Build it: sync and canary (70 min). Fix the sync, starting with a dry run the data owner checks. Add the canary pair to each collection. Schedule the weekly check with a line for every signal and a named person for each.
5. Build it: runbook and log (40 min). Write the four runbook sections and start the withdrawal log with any withdrawal you already found.
6. Build it: handover (40 min). Have your substitute run a canary withdrawal and an urgent withdrawal from the runbook while you watch without helping. Fix every place they got stuck.

Exit check

Passing test: a withdrawal drill record for one connected knowledge base, containing:

• the refresh and withdrawal windows, written before the drill;
• the canary pair: the current and the withdrawn marker, both found before the withdrawal, checked by an ordinary test account;
• the times: when the source removed the withdrawn marker, when each place you checked stopped showing it, and the current marker still found afterwards;
• the withdrawal log with at least one entry, including who received the document and how they were told, or that nobody did;
• your substitute's run: their name, the date, and where they got stuck.

It passes when the withdrawn marker is gone within the stated window in every place checked, the current marker still works, and your substitute ran the drill from the runbook alone.

Rule to remember

Deleting the file doesn't delete the answer. Check that the answer is gone.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T04-L04 · Connect an Assistant to Real Documents Without Widening Anyone's Access: the collections, groups and negative test this book keeps running.
• T04-L03 · Test and Fix the Document Assistant Your Group Relies On: checking answers are right, after every document change.
• T03-L05 · Keep a Production Prompt Healthy for Months, and Hand It Over: the same weekly signals and handover, for a prompt.
• T12-L05: governance records and handover across all AI systems.
• T15-L05 · Keep Your Group's AI Use Compliant and Running After You Leave: archiving and handover for research groups.

Tools

• Open WebUI: Taught. The department's assistant, its knowledge bases and audit log.
• Microsoft Copilot: Taught. The company's assistant, with audit records per answer.
• Onyx: Catalogued. The alternative with built-in retrieval logs.