Run Automatic Checks, an AI Reviewer and Releases You Can Roll Back

T08-L05 · Write code · Level 5 Operator · 20-minute read, plus 4 hours of doing and 15 minutes a week

Green, and wrong

The plate summary from T08-L04 now runs every night on the institute's server: it reads the LIMS and writes the summaries for five groups. Six people change it, about fifteen pull requests a month. A colleague set up an AI reviewer that comments on each one, and gave it his own access token "to save time".

One Monday, three groups' summaries are off by a few percent. Looking back, Mira finds three things. A pull request had added a small package for reading plate files, published two weeks earlier by an account nobody knew. The assistant that helped write the change had also written its tests, and they passed. And overnight, when a reviewer comment pointed out a failing check, the AI reviewer had pushed a "fix" to the branch itself, using the colleague's token. The server ran whatever was on main that night. Nobody could say which version had produced Friday's numbers.

In a company it's Leonie's returns report, now feeding the quality dashboard for three sites. The AI review bill doubled in a month because the reviewer ran on every small commit, and when a dashboard figure was questioned in an audit, nobody could show which checks the running version had passed.

You are at Level 5 Operator. Many people rely on this code every day. When it breaks, their numbers are wrong or their work stops. This book is about keeping shared code safe to change for months: what you watch, what you do when it breaks, who takes over, and what you can show an auditor.

After this you can

• Run automatic checks on every pull request and make them required before anything can be merged.
• Add an AI reviewer that reads and comments but can never push, approve or merge, with a cost limit.
• Decide on every new package with a short dependency note, and keep installed packages pinned and scanned.
• Release by version, keep a release record, and roll back to the previous version in minutes.
• Review the numbers weekly, handle an incident with a runbook, and hand the operation to a deputy.

Before you start

• Books: T08-L04: the protected main branch, pull requests, the assistant's rules and the repository record this book keeps running. T08-L03 for the spec and known-answer tests. T12-L05 for governance records across systems.
• A team repository on Gitea or GitHub with main protected, and admin rights on it or someone who has them.
• A server where the code runs for colleagues, and whoever in IT runs it.
• A named deputy with three hours in their calendar.
• About 4 hours to set things up, then 15 minutes a week.

Last verified 23 September 2026. Gitea has had built-in automatic checks, called Actions, since version 1.19. Claude Code's GitHub reviewer comes as an app whose standard permissions include writing code; you can create a narrower one. Details for experts at the end of this book.

The idea in one page

At Level 4 you made sure every change gets a second person. At Level 5 many people rely on the result, and changes keep coming for months. Four habits keep that safe.

Checks run on every pull request, and nobody can skip them. Continuous integration, usually shortened to CI, means a server runs your checks automatically whenever someone proposes a change. The checks are the ones you already trust: the known-answer tests from the spec, the secret scan, and a scan of installed packages. Branch protection then makes them required: main accepts a pull request only when every check has passed. Tests the author or the assistant wrote for the new feature are welcome, but the known answers people worked out by hand are what decide.

The AI reviewer gets an opinion, not a key. An AI reviewer that reads each diff and comments catches things people miss. It never needs to push, approve or merge, so it gets no rights to do so. Its access is read plus comment. It runs once per pull request, with a turn limit and a monthly spend limit. Its comments are advice. The required checks and the human reviewer decide.

Every new package is a decision. A dependency is a package your code installs from outside. It brings its own code, and often more packages behind it. So a new one needs a short dependency note: what it's for, who publishes it and since when, what it pulls in, and why the standard library isn't enough.

Installed packages are pinned to exact versions with checksums, a fingerprint of each package file, so the server installs exactly what was reviewed. A scan checks them for known security problems on every pull request and once a week.

The server runs versions, not main. A release is a tagged version you decided to put into use; a tag, as in T08-L03, is a fixed name such as v1.6 attached to one commit. The server runs a release, never whatever is on main tonight. A release record says, for each version, which commit it is, which checks passed, who reviewed it and when it went live. Rollback means switching the server back to the previous release. You practise it before you need it.

What you watch every week.

What stays yours: which checks are required, what the AI reviewer may do, which packages come in, and which version runs.

Worked example: one incident, four quadrants

The method comes from T01-L01. The lab uses Gitea; the company uses GitHub. The people and incidents are examples written for this book.

Lab: the nightly plate summary

Structure it. Mira, now the maintainer, writes: "Five groups depend on the nightly summary. Any change must pass the known answers before it can be merged, nothing but people can change the code, every package is a decision, and the server runs a version we can name and roll back in minutes. Paul must be able to run all of it when I'm away."

Understand it. She reconstructs the incident from the Git history and the server's logs. The pull request that added the package had passed review because the diff looked tidy and the new tests were green. The new tests had been written with the assistant, against the new code's own output. Nobody had run the hand-worked known answers from tests/expected/, because running them was left to each author. The server's nightly job started with git pull, so it ran main as it was at 1 a.m.

The package itself turned out harmless, but it didn't need to exist: the spec says standard library only, and the new code used one function from it that Python's built-in csv module already does. The AI reviewer's token was the colleague's personal one, with full rights to every repository he could reach.

That changes her problem. Back to Structure it: "The package wasn't the failure. The failures were that the known answers weren't required, the AI reviewer had write access, and the server ran main instead of a release. Fix those three, then look at the package rule."

Choose tools.

• Checks: Gitea Actions, the CI built into the institute's Gitea server. IT already runs the server, and it adds a runner, the machine that executes the checks, on a small virtual machine of its own.
• AI reviewer: OpenCode in its command-line form inside the check, with a read-only agent, because the group already uses OpenCode and its permissions file from T08-L04. The alternative is Claude Code, which has a ready-made GitHub reviewer; the company example uses that.
• Package scans: pip-audit, a free tool that checks Python packages against a public list of known security problems.
• Releases: no deployment platform. A release is a folder on the server and one link, which IT can read in a minute.

Build it: required checks (60 min). The workflow file in .gitea/workflows/checks.yaml tells the runner what to do for every pull request:

name: checks
on: [pull_request]
jobs:
  tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - name: Known answers, on a Python with no extra packages
        run: python -m unittest -v
      - name: Secret scan
        run: gitleaks git --no-banner

Read from the top: the file runs on every pull request, on a runner that offers a standard Linux system. The first step fetches the branch's code, the second sets up Python, and the last two run the known-answer tests and the secret scan. The runner's machine has gitleaks installed, as every contributor's laptop does.

The tests run on a fresh Python with nothing else installed. That enforces the spec's "standard library only" rule without anyone remembering it: an added package makes the tests fail. In branch protection she adds checks / tests as a required status check, so the merge button stays grey until it passes.

Then she proves it. She opens a test pull request that changes one rounding step and nothing else. The known-answer test fails, and the merge is blocked. She opens a second one that changes a value in tests/expected/ to make a wrong answer pass. The checks go green, which is exactly why the repository has a second rule: changes to tests/expected/ need the maintainer as reviewer, and the pull request template, the text Gitea puts into every new pull request's description, asks the author to say so. She keeps both pull requests, closed, as evidence.

Build it: the comment-only AI reviewer (45 min). The reviewer is an OpenCode agent defined in one file, .opencode/agents/reviewer.md. Its permissions deny every edit, every command and every web request, and it stops after 15 steps:

---
description: Comment-only reviewer for pull requests
mode: primary
permission:
  edit: deny
  bash: deny
  webfetch: deny
steps: 15
---
Read pr.diff and SPEC.md. Report three things only: code that no spec rule
asks for, changed behaviour that no test covers, and any new import.
Plain text. Never propose a fix as code.

It runs from a second workflow file, .gitea/workflows/review.yaml, in two steps that hold different keys:

• the model key for the institute's model gateway, available to the OpenCode step only;
• the comment key, belonging to a bot account review-bot with read access to this one repository, and a token limited to writing comments. It can't push, approve or merge.

name: ai-review
on:
  pull_request:
    types: [opened, reopened]          # once per pull request, not on every commit
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - name: Review (model key only)
        env:
          GATEWAY_KEY: ${{ secrets.MODEL_GATEWAY_KEY }}
        run: |
          npm install -g opencode-ai
          git diff origin/${{ github.base_ref }}...HEAD > pr.diff
          opencode run --agent reviewer "Review this pull request." > review.md
      - name: Post the review as a comment (comment key only)
        env:
          BOT_TOKEN: ${{ secrets.REVIEW_BOT_TOKEN }}
        run: |
          python3 -c "import json; print(json.dumps({'body': open('review.md').read()}))" > body.json
          curl -sf -X POST -H "Authorization: token $BOT_TOKEN" -H "Content-Type: application/json" \
            -d @body.json "${{ github.server_url }}/api/v1/repos/${{ github.repository }}/issues/${{ github.event.pull_request.number }}/comments"

The agent can't write files, so the workflow makes the diff file for it and saves what OpenCode prints as the review. The repository's opencode.json names the gateway as a provider and reads its key from the GATEWAY_KEY variable, as T08-L04's Details for experts describe.

Both keys are stored as Actions secrets: values saved in the repository's settings, under Actions, that a workflow can use but nobody can read back. Mira creates the comment key while signed in as review-bot: in its user settings, under Applications, she generates a token with only the issue scope set to write. Gitea treats a pull request's comments like an issue's, so that scope is enough to comment and nothing more.

The colleague's personal token is revoked. The AI reviewer runs when a pull request is opened or reopened, not on every commit, which keeps the cost at about ten runs a month.

Build it: the dependency rule (20 min). The contributing guide gets a short rule: a new package needs a dependency note in the pull request and the maintainer's approval. For this script the answer will almost always be no, and the fresh-Python check enforces that. The institute's other, larger repositories use the full version of the rule: packages pinned with checksums, a weekly pip-audit run, and a monthly pull request that updates everything at once, reviewed like any other change.

Build it: releases and rollback (60 min). With IT she changes how the server runs the script. A release is a tag on main, such as v1.6. A short deploy script, run by a person, copies that tag into its own folder on the server and points a link called current at it. The nightly job runs current, never main. Rolling back means pointing current at the previous folder: one command, and the previous version runs the next night.

#!/bin/sh
# deploy.sh v1.6 : put release v1.6 in its own folder and make it current. Run by a person.
set -e
cd /srv/plate-summary
git -C repo fetch --tags
git -C repo worktree add "../releases/$1" "$1"       # a separate folder holding exactly that tag
ln -sfn "releases/$1" current                         # point the link at it
echo "Now running $1"

Rolling back to v1.5 is the last line alone: ln -sfn releases/v1.5 current, run in the same folder.

Each release gets a line in RELEASES.md:

v1.6 · commit 4e1a9c2 · checks passed · reviewed by Paul · live 2 Oct 2026 · deployed by Mira
v1.5 · commit 9b07d3f · checks passed · reviewed by Mira · live 18 Sep 2026 · deployed by Paul

They practise a rollback once with IT watching: v1.6 to v1.5 and back, in six minutes, with a check that the nightly job used the right folder.

Build it: the weekly look, the runbook and the handover (45 min). Every Monday, 15 minutes: failed checks and any flaky ones, the oldest waiting pull request, the AI reviewer's cost and how many of its comments led to a change, and package alerts. One row in a shared spreadsheet.

The runbook has five steps. Stop: if last night's numbers look wrong, roll back to the previous release and tell the five groups which nights are affected. Find: the release record names the version and commit that ran. Fix: a pull request, with a new known-answer case that would have caught it. Tell: the group leaders; IT if the server or LIMS access was involved. Record: a line in the incident log.

Paul does the handover. He runs a Monday review alone, then handles a staged incident: Mira releases a copy with a deliberate rounding error on a test server, and he notices it, rolls back, finds the commit and writes the missing test. He gets stuck once, on where the deploy script lives. That goes into the runbook. The five group leaders get a one-line summary: the nightly summary now names its version at the bottom, and any version can be traced to its checks and reviewer.

What it taught her: none of the three failures was the AI's fault alone. Each was a place where the process relied on someone remembering, and each is now something the server refuses.

Company: the returns report on GitHub

Structure it. Leonie writes: "The returns report feeds the quality dashboard for three sites. It must pass its checks before every merge, the AI reviewer must stay within budget and never change code, and an auditor must be able to see which checks the running version passed."

Understand it. The AI reviewer had been installed through Claude Code's GitHub setup, with the standard Claude GitHub app. That app's permissions include writing code, running workflows and changing repository hooks, most of which the review doesn't use. It also ran on every push to a pull request branch, which is why the bill doubled. And the dashboard server pulled the report's code on start-up, so the audit question had no answer. Back to Structure it: "Narrow the reviewer, run it less often, and make the server run releases."

Choose tools.

• GitHub Actions for the checks, with pip-audit among them.
• The Claude Code GitHub action for review, but through a custom GitHub app that IT creates with only the permissions the action needs, and a workflow that gives the job read access plus the one tool that posts review comments.
• Dependabot, GitHub's built-in service that opens pull requests to update packages, set to once a month.
• Code owners, a file that names who must review changes to certain paths, so that changes to the known answers and to the package list always need Leonie or her deputy.

Build it. Required checks: known-answer tests, secret scan, pip-audit on the pinned package list. The reviewer runs when a pull request is opened or marked ready for review, with a turn limit. Its API key sits in a workspace of the Claude Console, Anthropic's account website: an area with its own keys and a monthly spend limit. The dashboard server runs tagged releases with a release record, as in the lab. A rollback drill with the ERP team watching takes nine minutes.

The auditor's question gets a one-page index: the branch protection settings, the required checks, the release record, the reviewer's permissions and budget, the dependency notes and the incident log. The next time a dashboard figure is questioned, Leonie answers from the release record in two minutes.

Sign-offs. IT for the custom app and the runner, the head of quality for the release process, and the ERP team for the rollback. Her deputy, Tim, runs the Monday review and a rollback drill before her holiday.

In both places the AI stayed useful and small. What made the code safe for many people was that the server now refuses what people used to have to remember.

What goes wrong

Checks that aren't required. Symptom: a red check, and the pull request merged anyway. Fix: make each check a required status check in branch protection, and apply it to administrators too.

Tests written by the change. Symptom: green checks and wrong numbers. Fix: the hand-worked known answers decide, and changes to them need the maintainer's review.

An AI reviewer with write access. Symptom: commits by the reviewer, or a personal token in its setup. Fix: read plus comment only, its own account or app, and the model key and the comment key in separate steps.

A reviewer that runs on everything. Symptom: the AI bill doubles and comments repeat. Fix: run it when a pull request opens or is marked ready, cap its turns, set a monthly limit, and check weekly whether its comments lead to changes.

Packages nobody chose. Symptom: a new package in a tidy diff, published last month by an unknown account. Fix: a dependency note and the maintainer's approval for every new package, pinned versions with checksums, and a scan on every pull request.

The server runs main. Symptom: nobody can say which version produced last Friday's numbers. Fix: releases by tag, a release record, and the version printed on every output.

A rollback nobody has tried. Symptom: the first rollback happens during an incident, and takes an afternoon. Fix: practise it with IT, and again every six months.

Do it yourself (4 hours, then 15 minutes a week)

1. Structure it (15 min). Write who depends on the code, what must be true before any merge, and who takes over when you're away.
2. Understand it (40 min). Go back through your last problem or near miss. What was checked, by whom, and which version ran? List every token and app with access to the repository.
3. Choose tools (15 min). Decide the CI, the AI reviewer's route and rights, the package rule and how the server runs releases.
4. Build it: checks (60 min). Write the workflow, make it required, and prove it with a pull request that breaks a known answer.
5. Build it: the AI reviewer (45 min). Read-only agent, separate comment key, runs once per pull request, turn and spend limits.
6. Build it: releases (60 min). Release by tag, record, rollback script. Practise the rollback with IT.
7. Build it: runbook and handover (45 min). Write the runbook. Your deputy runs a weekly review and a staged incident without you.

Exit check

Delivered artifact: a repository and its operations record, with:

• required checks and the closed test pull request they blocked;
• the AI reviewer's rights, keys, trigger and spend limit;
• the dependency rule and at least one dependency note;
• the release record and the rollback drill, with its time;
• four weeks of weekly reviews, the runbook and the handover record.

It passes when a pull request that breaks a known answer can't be merged, the AI reviewer can't push, approve or merge, a rollback restores the previous version in under 15 minutes, and your deputy has done a weekly review and a staged incident without your help.

Rule to remember

Let the server refuse what people would have to remember, and let the AI advise, never decide.

Details for experts

Checked on 23 September 2026. You don't need this to follow the book; it's for setting things up or answering a colleague.

Further reading and tools

Next books

• T08-L04 · Use OpenCode Desktop on Your Organisation's Real Repositories: protection, pull requests and the assistant's rules.
• T08-L03 · Write the Spec First, So an AI Assistant Can Change Code Your Team Depends On: the known answers the required checks run.
• T07-L05 · Keep a Team of Agents Running for Months, With Memory a Person Approves: the same weekly review and handover for agents.
• T12-L05 · Keep One List of Every AI System Your Team Runs, and Make It Survive You: records across every system your team runs.
• T12-L04 · Lock Down a Self-Hosted AI Service Before Real Data Goes In: the server and runner.

Tools

• Gitea: Taught. Actions, required checks and scoped tokens in the lab example.
• GitHub: Taught. Actions, code owners and Dependabot in the company example.
• OpenCode: Taught. The lab's comment-only reviewer.
• Claude Code: Taught. The company's reviewer through its GitHub action.
• Git: Taught. Tags and the release record.
• Claude Sonnet: Catalogued. A model for the reviewer.