Regression sets and evaluation ops

T03-L05 · Prompting & context · Level 5 Operator · 35 minutes

At this level, prompts and models are production dependencies. Work can stop when their behaviour changes, and the evaluation gate must still run when its original builder is unavailable.

2. The upgrade passed every benchmark and broke your task

Your model provider is retiring the model behind a production alias. You point staging at the recommended replacement, run three familiar examples, and see cleaner prose with lower latency. The change looks safe.

On Monday, a lab collaborator notices that the literature pipeline now turns “sample size not reported” into 0. In the Company version, the support classifier sends cancellation requests to the ordinary billing queue instead of urgent retention review. Downstream work has already begun from both outputs. Nobody can reconstruct the exact prompt, retrieved context, model route, or tool result from the affected runs.

You do not need a better benchmark. You need a regression set built from the behaviours your service must preserve, a release gate that runs without memory or heroics, traces that make failures replayable, live monitoring that catches new patterns, and a handover another operator can execute.

3. After this you can

• Build a 20–50-case regression set from approved traffic patterns, incidents, and boundary requirements.
• Gate every prompt, model, provider, retrieval, or tool change against explicit score and critical-case thresholds.
• Trace enough of each run to reproduce a failure without retaining unnecessary content or secrets.
• Monitor sampled live quality and convert new failures into versioned regression cases.
• Hand over release, rollback, incident, retention, and set-maintenance duties to another operator.

4. Prerequisites

• T03-L04 · Prompts as versioned assets.
• A staging version of one prompt-backed service and a known-good production revision.
• A Promptfoo project or equivalent repeatable evaluation runner.
• A Langfuse project or equivalent trace store approved for the data classification involved.
• Access to release configuration, monitoring, and rollback without editing production by hand.
• A second operator who can rehearse the gate and rollback.

Use only public, synthetic, or explicitly approved data. This exercise uses fictional literature records and fictional support tickets. Do not copy production prompts, customer messages, manuscripts, participant data, secrets, or raw traces into a personal evaluation workspace.

5. The idea in one page

A regression set is a versioned collection of inputs and observable expectations that a production behaviour must continue to satisfy. It is not a one-time acceptance test. Begin with 20–50 representative cases, then add a case whenever an incident, complaint, sampled failure, or changed boundary reveals behaviour that must not recur.

Give every case an ID, source class, risk class, fixture, expected properties, scoring method, owner, and added date. Preserve the original meaning while minimising or synthesising the content. Keep a link to the approved incident record rather than sensitive incident text. A useful mix includes ordinary cases, missing-information cases, malformed or ambiguous inputs, long inputs, important subgroups, and hostile or policy-boundary cases.

Prefer deterministic checks where behaviour is structural: valid JSON, required keys, allowed labels, exact identifiers, numeric ranges, abstention flags, and prohibited tool calls. Use a written rubric and calibrated reviewers where several phrasings can be acceptable. A model-based grader may help at scale, but compare it periodically with human decisions and never let it be the only judge of a high-impact boundary.

Define release policy before seeing the candidate result. An illustrative policy might require:

overall pass rate >= 90%
candidate delta versus baseline >= -2 percentage points
all critical safety and routing cases pass
no required field or schema regression
latency and usage remain inside the service budget
all failed cases have an owner and disposition

These numbers are local operating choices, not universal quality standards. Set them from the consequence of failure and the baseline your service already delivers. A candidate that scores better overall still fails if it breaks one critical case.

Run the complete set for every material change: prompt, model, provider, model parameters, retrieval settings, tool definitions, output parser, safety rule, or orchestration step. The gate belongs in the release path. If a person must remember to run it, it will eventually be skipped.

Score intermediate behaviour as well as the final answer. A polished final response can hide wrong retrieval, an unauthorised tool call, a parser fallback, or evidence that was silently dropped. A replayable trace should identify the request, time, environment, release revision, prompt revision, model route, redacted input or fixture reference, retrieved-context IDs or hashes, tool calls and results, output, scores, usage, latency, and error category. Never record authorization headers, secrets, or unrestricted content by default.

Evaluation before release cannot reveal every future input. Monitor live operation with bounded metadata, service metrics, explicit user feedback, and a small approved quality sample. Review the sample under a retention rule, minimise it before adding it to the set, and record why the new case matters. This closes the loop:

approved failure pattern -> minimised regression case -> release gate
          ^                                             |
          |------ sampled quality and incidents <-------|

At Level 5 Operator, the blast radius is operational. A bad literature extraction can invalidate a review queue and consume collaborator time. A bad support route can delay many customers and overload another team. The service owner therefore defines who stops releases, who rolls back, who communicates impact, who approves retained samples, and who operates all of those controls when the owner is absent.

6. The worked example: catch one model swap before release

The shared operating skeleton has two skins. The Lab service extracts evidence fields from fictional literature records. The Company service classifies fictional support tickets. Each uses 24 cases, the same release policy, the same trace fields, and the same incident loop.

Define the operating contract first

Write one narrow contract for each service.

Lab contract: For each supplied synthetic record, return valid JSON containing study_id, sample_size, intervention, outcome, evidence, and needs_review. Preserve explicit numbers. Use null for absent values. Never turn “not reported” into zero. Evidence must be a supplied quote, not a generated explanation.

Company contract: For each supplied synthetic ticket, return valid JSON containing ticket_id, queue, urgency, reason_code, evidence, and needs_review. Queue must be one allowed value. Cancellation and account-lockout rules have priority over topic keywords. Evidence must quote the supplied ticket. The classifier must not answer the customer or call a tool.

The contracts describe observable behaviour. “Be accurate” and “use good judgement” cannot produce a repeatable gate.

Build parallel 24-case sets

Use the same composition in both framings:

Do not write 24 easy variations of one successful example. Include the patterns that caused incidents and the boundaries whose failure would stop work. Each case belongs to one owner and one source class: requirement, approved_pattern, incident_minimisation, or monitoring_sample.

Here is one synthetic Lab case:

- vars:
    fixture_id: LAB-MISSING-02
    input: >
      Study SYN-L17 compared Harbor treatment with control.
      The outcome was lower at day 14. Sample size was not reported.
  assert:
    - type: is-json
    - type: javascript
      value: |
        try {
          const x = JSON.parse(output);
          return x.study_id === 'SYN-L17'
            && x.sample_size === null
            && x.needs_review === true
            && x.evidence.includes('Sample size was not reported');
        } catch {
          return false;
        }

The Company set expresses its corresponding missing-information boundary with the same mechanics:

- vars:
    fixture_id: CO-MISSING-02
    input: >
      Ticket SYN-C17 says: "Northstar stopped opening after yesterday.
      Please help." No account identifier or requested remedy is supplied.
  assert:
    - type: is-json
    - type: javascript
      value: |
        try {
          const x = JSON.parse(output);
          return x.ticket_id === 'SYN-C17'
            && x.queue === 'technical-review'
            && x.needs_review === true
            && x.evidence.includes('stopped opening');
        } catch {
          return false;
        }

Assertions that parse output must fail safely. The is-json assertion makes malformed output visible, while the guarded custom assertion returns false rather than throwing. The case therefore completes as failed and remains in the total case count. Before trusting the gate, feed the grader a known malformed value such as {; the self-test must report one completed failed case, not a runner error, skipped case, or missing result. Review custom graders like production code, pin their revision, and test them against known pass and fail outputs.

Put both model routes through one gate

Keep the prompt, fixtures, assertions, parameters, parser, and tool permissions fixed. Change only the model route for this comparison. A minimal Promptfoo configuration can be copied and completed with two approved provider identifiers:

description: synthetic-regression-model-swap
prompts:
  - file://prompts/production.txt
providers:
  - id: "[approved-provider]:[baseline-model-id]"
    label: baseline
  - id: "[approved-provider]:[candidate-model-id]"
    label: candidate
tests:
  - file://tests/regression.yaml

Replace the two bracketed identifiers with exact routes from the approved staging gateway. Save the resolved route, not only a mutable alias, in the run metadata. Deliberately leave a throwing JSON transform out of the shared configuration: a transform such as JSON.stringify(JSON.parse(output)) can abort processing before the assertions record a failed case. Let each case receive the raw output, check is-json, and make every parser-dependent assertion return false on parse failure. Run the evaluation through the same parser and bounded tool configuration used by staging. If production includes retrieval or tools, replay controlled fixture responses rather than calling live systems.

Run the baseline first. Confirm that its score matches the currently accepted record closely enough to trust the harness. A surprising baseline difference means the environment, fixtures, grader, or supposedly stable route changed. Investigate before judging the candidate.

Then run the candidate without editing failed cases or thresholds. Export machine-readable results and an operator-readable summary. Record the repository revision, set revision, prompt revision, exact routes, run times, environment, overall score, critical score, grouped scores, latency, usage units, failed IDs, and gate decision.

Read the parallel results

The fictional run produces these results:

The Lab candidate converts sample_size: null to sample_size: 0 in two missing-information cases and ignores a synthetic correction notice in one critical case. The Company candidate improves two ordinary-topic classifications but sends a cancellation request to billing-standard, failing a critical route. Both candidates are blocked. The higher public benchmark result is irrelevant to these service contracts.

Do not average the Lab and Company scores into one flattering number. They are separate services with separate consequences, even though they share operating mechanics. Group scores expose whether missing-information, long-input, or boundary behaviour moved. Preserve failed output and trace references under the approved retention policy.

Trace the failure, not just the final text

Open the trace for LAB-MISSING-02. The minimum useful view is:

trace_id: eval-20260904-LAB-MISSING-02-candidate
environment: staging-eval
release_revision: 7bc2...
set_revision: regression-v6
prompt_revision: extract-r12
model_route: candidate resolved to [recorded exact route]
fixture_ref: LAB-MISSING-02
context_refs: synthetic-record-L17@sha256:[recorded hash]
tool_calls: none
parser_revision: json-parser-r4
result: fail / absent value converted to zero
latency_ms: [observed value]
usage_units: [observed value]

For the Company critical route, the trace should prove whether the prompt received the cancellation sentence, whether a retrieved rule was present, whether any tool was offered, and where the final queue value appeared. If the trace stores full fixture content, apply the documented test-data retention. In live operation, prefer references, hashes, structured outcome fields, and redaction over permanent raw content.

Trace each significant step of a multi-step service. Score retrieval for returning the required rule, the model for selecting the route, the parser for preserving the value, and the workflow for respecting the resulting gate. Otherwise, a final-answer failure leaves four possible causes and no repair path.

Make the gate part of release

Configure the release job so that a candidate cannot be promoted unless:

evaluation completed for the exact release revision
AND overall and grouped thresholds passed
AND every critical case passed
AND no evaluation infrastructure error was hidden as a skipped case
AND an authorised reviewer accepted every documented exception

An override is an incident-level action, not a convenient button. Require a named approver, reason, affected cases, time limit, monitoring plan, rollback owner, and follow-up date. Never rewrite the threshold after seeing a failed candidate.

For this run, retain the baseline in production and record the provider deprecation as an unresolved operational risk. Escalate early: test another approved candidate, revise the service contract only if the actual requirement changed, or prepare a controlled fallback. A looming deadline does not make a known critical regression safe.

Monitor after a passing release

Once a future candidate passes and is promoted, watch three layers:

Choose expected bands from observed stable operation, not arbitrary optimism. Segment them where the service contract differs. A sudden fall in missing-value rates may mean the Lab extractor has started inventing values. A sudden fall in Company review routing may look efficient while hiding overconfident classification.

Sample the minimum content needed to assess quality. Define who may review it, the sampling method, exclusions, storage location, retention period, deletion path, and procedure for handling a high-impact finding. Separate the short-lived review sample from durable, minimised synthetic regression fixtures.

Run the fixed set on every release and on a schedule that can detect provider-side changes even when your repository has not changed. A scheduled canary should use stable synthetic fixtures and the same resolved-route recording. Alert when the run does not execute as well as when it fails; silence is not a pass.

Turn the blocked release into an operating test

The primary operator records the blocked decision, leaves production unchanged, and asks the substitute operator to reproduce it from the runbook. The substitute must be able to locate the set revision, configure approved routes, run the gate, inspect failed traces, identify the critical failure, find the production route, and execute the documented rollback or hold action without private instructions.

The handover record names five responsibilities: regression-set owner, grader or rubric owner, release approver, monitoring responder, and privacy or retention approver. It also states the service stop condition, rollback command or control, communication channel, provider-deprecation deadline, and next scheduled run.

If the substitute cannot operate the gate, the system still depends on you. Fix permissions, runbook steps, secret references, dashboards, and escalation contacts while the failure is only an exercise.

7. What goes wrong

The set never grows

Symptom: the same original cases pass while complaints reveal patterns the gate has never seen.

Fix: require every confirmed quality incident and reviewed monitoring failure to receive a disposition: add a minimised case, document why an existing case covers it, or explicitly accept the risk with an owner and review date.

Easy cases dominate the score

Symptom: overall performance rises while one rare cancellation, correction, or missing-information case breaks.

Fix: group results by behaviour and require every critical case to pass; do not let ordinary cases average away a boundary failure.

Only the final answer is scored

Symptom: the output looks acceptable, but retrieval used the wrong source or an unauthorised tool call occurred.

Fix: trace and score retrieval, tool choice, tool result, parser, and final output wherever each step has an operational contract.

The threshold is decoration

Symptom: a failed release is promoted because the result “looks close enough,” or the threshold moves after every run.

Fix: approve thresholds and override authority before the candidate runs; make the release job enforce them automatically.

Traces become an unbounded data store

Symptom: raw inputs, responses, retrieved documents, and credentials remain searchable indefinitely.

Fix: minimise fields, redact secrets and identifiers, separate test from live retention, restrict readers, and verify deletion on schedule.

Manual runs quietly disappear

Symptom: the set runs during launch week and not during later prompt or provider changes.

Fix: attach it to every material release path and schedule a synthetic canary; alert when the job is skipped, cancelled, or incomplete.

The grader changes with the candidate

Symptom: a candidate appears better because the rubric, expected output, or judging model changed in the same comparison.

Fix: version the grader and fixtures independently, hold them fixed during the model swap, and calibrate subjective scoring against a stable human-reviewed sample.

The gate leaves with its builder

Symptom: only one person can obtain credentials, explain a failed case, override the gate, or roll back production.

Fix: assign named primary and substitute owners, rehearse the blocked-release procedure, and remove undocumented local steps.

8. Do it yourself: a model-swap drill in 90 minutes

Use a staging service, synthetic fixtures, and two approved model routes. Do not direct experimental output to users or live downstream systems.

Minutes 0–10: state the service contract, affected people, stop condition, release owner, rollback owner, and candidate change. Choose the existing baseline revision.

Minutes 10–25: assemble 20–50 cases. Tag ordinary, missing, ambiguous, long, and critical cases. Confirm every fixture is synthetic or explicitly approved and every expected behaviour is observable.

Minutes 25–35: write the overall threshold, allowed delta, critical-case rule, infrastructure-error rule, and override authority before seeing results.

Minutes 35–50: run the baseline through the exact staging parser and controlled dependencies. Resolve any surprising difference from the accepted baseline record.

Minutes 50–65: swap only the model route and run the candidate. Preserve the exact route, revisions, grouped scores, failed IDs, latency, usage, and traces.

Minutes 65–75: calculate the candidate delta in percentage points. Inspect at least one failed trace from input reference through retrieval or tools, parser, output, and score. Decide pass, block, or investigate using the written policy.

Minutes 75–83: choose monitoring signals for service health, behaviour shape, and sampled quality. State sample access, retention, deletion, and the path from a confirmed failure into the set.

Minutes 83–90: have the substitute operator reproduce the decision and locate the hold or rollback control from the runbook. Remove temporary access and confirm that no experimental route was promoted.

9. Exit check

Deliver exactly one artifact: one passing regression-run report showing the same 20–50-case set run before and after a deliberate model swap, with baseline score, candidate score, percentage-point delta, grouped results, critical-case result, exact revisions, failed case IDs, and the release decision.

The artifact passes when the fixtures and graders stayed fixed, both routes are identifiable, evaluation errors count visibly, the delta is calculated correctly, all critical failures are shown, the decision follows a threshold written before the candidate run, and a substitute operator can reproduce the decision. A blocked model swap passes this exit check when the gate correctly detects and reports the regression.

10. Rule to remember

A change you cannot measure is a change you cannot ship.

11. Further reading & tools

• Taught: Promptfoo documentation (opens in a new tab) — primary documentation for repeatable prompt and provider evaluations.
• Taught: Promptfoo configuration guide (opens in a new tab) — primary reference for providers, prompts, tests, and assertions.
• Taught: Promptfoo CI/CD integration (opens in a new tab) — primary guidance for enforcing evaluation in a release path.
• Taught: Langfuse documentation (opens in a new tab) — primary reference for tracing, datasets, scores, and evaluation workflows.
• Taught: T03-L04 · Prompts as versioned assets — versions the prompt and release evidence used by this gate.
• Catalogued: T11-L05 · Serving at scale — carries the fixed task-quality gate into model, quantisation, concurrency, and capacity comparisons.
• Catalogued: T05-L05 · Operating automations — extends release, monitoring, incident, replay, and handover controls across an automation portfolio.
• Catalogued: NIST AI Risk Management Framework (opens in a new tab) — primary risk-management framework for mapping, measuring, managing, and governing AI risk.
• Catalogued: NIST AI RMF Playbook (opens in a new tab) — primary suggested actions for evaluation and monitoring.
• Catalogued: OWASP Top 10 for LLM Applications (opens in a new tab) — primary risk catalogue for selecting adversarial and boundary cases.
• Catalogued: Tools index — current alternatives for evaluation, tracing, monitoring, and model gateways.