Decide Which Problems Deserve Building | Heidelberg AI Curriculum
T17-L05
Decide Which Problems Deserve Building
Put every AI project your lab or company runs or wants on one sheet, as problems. Score each by value, effort, risk and who keeps it running after you leave, retire what no longer earns its upkeep, and keep a decision log a successor can follow without you.
T17-L05 · Structure & understand · Level 5 Operator · 17-minute read, plus 4 hours of doing
Eight things, one person, four months
Your group now runs more AI than anyone planned. The bench-note assistant for the sample inventory is in daily use. There's a weekly literature digest, an image-analysis script for the confocal data, a tool that checks figure legends against the methods, and a folder of saved prompts for reviewer replies. Proposed: a grant-writing helper, an extension of the inventory to clinical samples, and a chatbot for the core facility's booking questions. The PI, the group leader, asks at the group retreat which of these you'll still have next year. Then someone points out that the postdoc who built three of them leaves in March, and nobody else knows how the image script runs.
In a company it's already happened. The invoice-matching automation, built by a working student, a student employed part-time, two years ago, stopped in August when the bank changed its export format. Accounts payable matched invoices by hand for three days while someone found the login. Meanwhile the director wants a chatbot on the website, sales wants AI quotes for instruments too, and the contract-deadline list needs someone to keep it current.
You are at Level 5 Operator. When something you run breaks, work stops, and it must survive you leaving. The question now is not whether an idea is good. It's which of many your team can build and keep running without you. This book gives you one sheet, four scores and a decision log to answer that.
After this you can
List everything your group or team runs or wants to build on one sheet, as problems rather than tools.
Score each item on value, effort, risk and who will keep it running after you leave.
Decide build, keep, hold or retire for each item, within the hours your team can actually spend.
Retire something deliberately, with its data, access and reason taken care of.
Keep a decision log that a colleague who never met you can follow.
A shared spreadsheet with two tabs, Portfolio and Decisions, stored in a team or group drive that belongs to the organisation, not in your personal drive. When your account is closed, the sheet has to stay. Pick the place IT backs up and a new colleague gets access to on day one: a SharePoint team site, a Google shared drive, the group's wiki.
A chat assistant your organisation approves for internal material. Claude is a good pick for the two jobs in this book, arguing against your ranking and testing whether the log makes sense to a stranger, because it reads a long table closely. ChatGPT or Microsoft Copilot work too.
The problem statements, exploration notes and pilot results you have, and anyone's notes on what's running.
A 60-minute meeting with the people who decide (the PI, the head of department, the director) and the people who keep things running.
About 4½ hours: 17 minutes to read, about 4 hours to do, over two weeks.
The idea in one page
List problems, not tools. Every row is a problem someone signed, or would sign, in the form of T17-L02 · Frame a Problem Your Team Agrees On: who, what's hard, what a useful result looks like. "Image-analysis script" becomes "two students quantify confocal images for their theses; by hand it takes a day per experiment." Listing problems stops you ranking tools against each other, and it shows when two rows solve the same problem.
List what runs as well as what's proposed. Running things cost hours every month and break without warning. Proposals only cost hours once you start them. A portfolio that lists only proposals will overcommit the team every time.
Score four things, 1 to 5. Use whole numbers and write one line of reasoning per score.
Score
1 means
5 means
Where the number comes from
Value
Few people, rarely, small cost today
Many people, weekly, real cost or risk today
The problem statement's who and what's hard; pilot results
Effort
Hours to build and minutes a month to run
Months to build, days a month to run
Build estimate plus a year of running: fixes, updates, questions
Risk
A mistake stays on one desk and is easy to undo
A mistake reaches patients, customers, money or a real system, and is hard to undo
The blast radius (how far a mistake spreads) in the pilot plan; incidents so far
Keeper
Only you can run or fix it
A named keeper and a backup, written instructions, runs on systems the organisation looks after
Ask: if you left tomorrow, who restarts it?
The keeper score is a gate, not a weight. An item with Keeper 1 or 2 can't be decided build or keep until it has a named keeper and a backup. That rule does more for an Operator than any formula. It is how things survive you leaving.
Four decisions. For each row, one word:
Build: start next. High value, a keeper, and effort that fits.
Keep: it runs and earns its upkeep. Confirm the keeper, the backup and the hours.
Hold: not now. Write what would change the decision and when you'll look again.
Retire: switch it off, deliberately. Its data, its access and its reason are dealt with.
Decide within your hours. Add up the monthly running hours of everything marked keep, plus the build hours of everything marked build. If that is more than your team can spend, something moves to hold or retire. A portfolio without a limit is just a wish list.
Log every decision with its reason. A decision without a reason gets reopened by the next person who arrives. Each log entry has: date, item, decision, options considered, the reason, who decided, what would change our mind, and where the details are (problem statement, pilot note).
Where an assistant helps. It can't know what's valuable to your group, so don't let it score. It is good at two things. Give it the sheet and ask it to argue against your ranking: where are scores inconsistent, which low-ranked row has the strongest case? And give it only the decision log and ask the questions a successor would ask. What it can't answer from the log, a successor can't either.
In the four quadrants this is Structure it ⇄ Understand it at the level of the whole group or team: you structure a list, gather what's true about each row, score, challenge and rewrite. Choose tools and Build it then happen for the rows marked build, one at a time.
Worked example: one sheet, one meeting, one log
Both versions follow the same steps. Follow the one closer to your work. The rows, scores and decisions are typical examples written for this book.
Lab: eight projects and a postdoc who's leaving
Structure it: list the problems (45 min). The group's lab manager takes this on. She writes one row per running or proposed item, each as a problem, and adds the current keeper and the monthly hours.
Understand it: gather what's true (60 min, over a week). For each running item she asks the keeper three questions: how many hours a month, what broke this year, and who else could fix it. For each proposal she finds the problem statement, or writes down that there isn't one. Two things surprise her. The figure-legend checker has been used twice since spring. And the literature digest, which everyone reads, runs from the leaving postdoc's personal account.
Structure it: score and rank (45 min).
Problem (short)
Status
Value
Effort
Risk
Keeper
Hours/month
Proposed decision
Record samples at the bench (inventory assistant)
Running
5
2
3
4 (technician, backup lab manager)
3
Keep
Extend inventory to clinical samples
Proposed
4
2
4
4 (same)
+1, plus 15 to build
Build
Weekly digest of new papers for the group
Running
4
1
1
1 (leaving postdoc, personal account)
1
Keep, once it has a keeper
Quantify confocal images
Running
4
3
3
1 (leaving postdoc)
4
Retire, move to the core facility's maintained pipeline
Check figure legends against methods
Running
1
1
1
2
1
Retire
Saved prompts for reviewer replies
Running
3
1
1
5 (shared folder, anyone)
0
Keep
Grant-writing helper
Proposed
2
3
4 (unpublished data)
1
?
Hold
Core facility booking chatbot
Proposed
3
4
2
1 (not our system)
?
Hold, pass to core facility
The group can spend about 10 hours a month on keeping things running, plus one build at a time. The Keep rows come to 4 hours a month (3 + 1 + 0). The clinical extension adds 15 hours of build over two months, and then 1 hour a month, which brings running to 5. It fits.
Understand it: challenge (20 min). She pastes the table, with names replaced by roles, into Claude:
Here is our group's portfolio of AI projects, scored 1-5.
Don't re-score. Tell me:
1. Where scores look inconsistent between rows, and why.
2. The strongest case for the lowest-ranked item.
3. Which "Keep" or "Build" decision depends most on one person.
4. Anything our hours total leaves out.
Three answers are worth keeping. The image-analysis row has the same value as the digest but is retired, and the reason isn't on the sheet. It should be: the core facility runs a maintained pipeline that does the same job. The clinical extension depends on the data protection officer's pending answer, so the decision needs a condition. And the hours leave out time to hand things over before March. One answer isn't: it argues for the grant helper on value, without knowing the group writes one grant a year.
Structure it: decide and log (60-minute meeting). The PI, the lab manager, the technician and the leaving postdoc go through the sheet row by row. Two decisions change: the digest moves to the group's account this month, with the lab manager as keeper, and the image pipeline gets a date for the switch-over. Then the log. Two of the entries:
2026-10-14 · Quantify confocal images · Retire.Options: keep the postdoc's script with a new keeper; move to the core facility's maintained pipeline; stop quantifying.
Reason: same results on three test datasets; the core facility maintains its pipeline and backs it up; nobody else in the group can maintain the script.
Decided by: PI. Keeper until retired: postdoc, until 28 Feb.
What would change our mind: the core pipeline can't handle the new 16-bit files.
Details: comparison note in /Portfolio/confocal.
Retirement: results and script archived in the group drive; the postdoc's server job switched off on 28 Feb; students told on 1 Nov.
2026-10-14 · Grant-writing helper · Hold.Reason: one grant a year; drafts would need unpublished data in a tool that isn't approved for it.
Decided by: PI. Look again: when the institute approves an assistant for unpublished data, or at the April review.
Understand it: the successor test (20 min). She gives Claude only the Decisions tab and asks: "You joined this group today. Why did we stop quantifying images ourselves? Who fixes the digest if it breaks? When will we look at the grant helper again?" It answers all three. Then she asks a PhD student who wasn't in the meeting the same questions, with the sheet and nothing else. He answers two; the third, who fixes the digest, needs the backup's name, which was missing. She adds it.
Choose tools and Build it (briefly). One row is marked build: the clinical extension. It goes through its own trip round the quadrants, starting from its problem statement, with a pilot under T17-L04 · Run a Pilot with Real Users and Real Data. Retiring the image script, with data, access and accounts, follows T16-L05 · Keep Your Toolset Useful. A quarterly review goes in the group calendar, assigned to the lab manager's role, not her name.
Company: seven items and a three-day outage
Structure it: list the problems. The office manager lists seven rows: quote drafts in the CRM, the company's customer database (running), the request tracker in the CRM (running), the contract-deadline list (running), invoice matching (running), AI quotes for instruments (proposed), a website chatbot (proposed), and a meeting-notes summariser in the approved assistant (running, informally).
Understand it: gather what's true. The August outage is the first line in the invoice row: three days of manual matching, about 20 staff hours, a supplier discount missed. The keeper is nobody. The working student's documentation is a two-line note. The website chatbot has no problem statement. When she asks the director, the problem turns out to be "customers can't find the safety data sheets", which a better page on the website would solve.
Structure it: score and rank. Invoice matching: Value 5, Effort 2, Risk 4 (money, and a stop in accounts payable when it breaks), Keeper 1. By the gate it can't be keep until it has a keeper. Instrument quotes: Value 4, Effort 3, Risk 4 (larger amounts and configured prices), Keeper 3. Website chatbot: Value 2 for the stated problem, Effort 4, Risk 4 (customer-facing and unchecked answers), Keeper 1.
Understand it: challenge. The assistant points out that the instrument-quote row borrows its value from the consumables pilot, but instrument quotes are rarer and more complex, so it needs its own pilot. It also points out that three rows depend on the CRM administrator, who appears as backup everywhere.
Structure it: decide and log. The director signs:
Invoice matching: Keep, on one condition. Before anything new starts, someone in accounts payable becomes its keeper with IT as backup. The working student's setup is documented and moved to a company account, and a test with last month's bank file is run each quarter.
Instrument quotes: Build, after its own pilot. Starts in the next quarter, when invoice matching has a keeper.
Website chatbot: Hold. The reason and the trigger are logged: a better data-sheet page first; look again if customer questions about data sheets don't drop within three months.
Meeting-notes summariser: Keep, as personal use. It sits under the approved assistant's usage rule and needs no keeper of its own.
A second backup is named for the CRM rows, so not every CRM question lands on the CRM administrator.
In both cases the most important decision wasn't which new thing to build. It was which existing thing needed a keeper before anything new started. The gate made that visible.
Retire on purpose
Retiring is a decision, so it goes in the log like any other. Then four steps, in this order:
Tell the people who use it, with a date and what replaces it.
Keep what matters: export the data and results people need, and store them where the organisation keeps records.
Write why, in the log, with what would make it worth reopening. That line stops the same idea coming back every year without new information.
What goes wrong
Scores as false precision.Symptom: a long discussion about whether a row is 3.6 or 3.8. Fix: use whole numbers, 1 to 5, with one line of reasoning each. Discuss only rows where people's scores differ by two or more.
Only proposals on the sheet.Symptom: the team agrees to build two new things and then spends the quarter fixing old ones. Fix: list everything running, with its monthly hours, before you add a single proposal.
The keeper is you.Symptom: your name is in the keeper column of most rows. Fix: apply the gate. A row with only you as keeper is decided hold until someone else can run it, and handing it over becomes this quarter's work.
Nothing ever retires.Symptom: every review adds rows and none leave. Fix: at each review, ask of every running row: if we didn't have this, would we build it today? If the answer is no, it's a candidate for retirement.
Decisions without reasons.Symptom: the log says "Hold" and a year later nobody knows why, so the idea comes back as new. Fix: every entry gets a reason and what would change your mind. Run the successor test before you close the meeting.
The sheet lives with you.Symptom: the portfolio is in your personal drive, the review is in your calendar, and both vanish when you leave. Fix: store the sheet in a drive the organisation owns, and give the review to a role, with a named substitute.
The assistant ranks for you.Symptom: a neat ranking with confident reasons that nobody in the room recognises. Fix: the scores come from your problem statements, pilots and keepers. Use the assistant only to challenge the ranking and test the log.
Do it yourself (4 hours, over two weeks)
Structure it: list (45 min). In a shared spreadsheet in a drive that belongs to the organisation, list every running and proposed item as a problem, with its current keeper and monthly hours.
Understand it: gather (60 min). Ask each keeper the three questions: hours a month, what broke this year, who else could fix it. Find or note the missing problem statement for each proposal.
Structure it: score (45 min). Score value, effort, risk and keeper, one line of reasoning each. Apply the keeper gate. Propose a decision per row and check the hours against what the team can spend.
Understand it: challenge (20 min). Give the sheet, without names, to your assistant with the challenge prompt. Keep what fits what you know; note what you drop.
Structure it: decide (60-minute meeting). Go through the sheet with the people who decide and the keepers. Write the log entries during the meeting, not after.
Understand it: successor test (20 min). Give only the log to the assistant, then to a colleague who wasn't there, with three questions. Fix every gap.
Put the next review in the calendar, assigned to a role, three months ahead.
Exit check
Written finding: a ranked portfolio and a decision log.
It passes when:
every running and proposed item appears as a problem, with four scores, one line of reasoning each, and a decision;
at least one item is decided hold or retire, with a reason and what would change the decision;
every build or keep item has a named keeper and a backup, and neither is you alone;
the sheet and the log live in a place the organisation owns, and the next review is in a role's calendar;
a colleague who wasn't involved can answer three questions about past decisions (why did we stop X, who fixes Y, when do we look at Z again) from the log alone.
Rule to remember
Build only what someone other than you will keep running.
The RICE Scoring Framework - Overview, Example, and Explanation
The RICE Scoring Framework - Overview, Example, and Explanationairfocus by Lucid
T17-L05: Learn explicit prioritisation through Reach, Impact, Confidence and Effort. This 2020 RICE overview illustrates criterion-based decisions, not the book's scoring method. Retain risk, keeper and capacity gates and the decision log; airfocus adoption is unnecessary.
