▸How-to Scientist +1
A number you trust before you let the extractor loose on 5,000 papers — and a list of exactly which fields it gets wrong.
Track 17 · Advanced
17.4
Evals & testing
Evals (evaluations): measure whether your AI actually works — beyond a vibe-check
1Overview
An eval is a repeatable test that tells you whether your AI is getting better or worse — the difference between "it felt good in the demo" and "it is right 94% of the time, and here are the 6% it gets wrong." Taught as weak → strong pairs across the eval loop: read real failures (error analysis), grade them automatically (code · LLM-as-judge · human), validate the judge against human labels, and lock fixes in with a regression suite. Grounded in the methodology Hamel Husain & Shreya Shankar teach and Anthropic's own eval guidance, with a copy-ready cheat sheet, persona examples, and a primary-sourced FAQ.
This chapter teaches how to construct reliable evaluations that reveal the true performance of LLM‑based systems rather than superficial metrics. You will learn to design precision tests for extraction, detect hallucinated citations, analyze summarisation failures, and assess classifiers on rare but critical classes. The material also covers building regression suites, binary outcome judges, and cost‑quality tradeoffs so you can confidently deploy models only after they meet concrete, human‑validated standards.
1.1After this chapter you can
→Find what to measure by reading real failures (error analysis) — not guessing
→Pick the cheapest grader that works: code, an LLM-as-judge, or a human
→Default to binary pass/fail, and validate the judge against human labels
→Lock fixes in with a regression suite that runs on every change
1.2Why need repeatable tests?
Repeatable tests let you track real‑world success rates over time, showing the difference between a demo that feels right and an AI that is correct 94% of the time with known failure cases.
1.3How are failures graded?
You read actual errors, then grade them automatically using code, LLM‑as‑judge, or human judges, validate those judgments against human labels, and lock in fixes through a regression suite.
1.4The moves — weak → strong ladder
- 1Read real traces before you build any metric
- 2Group the notes into a failure taxonomy
- 3Only measure failures you have actually seen
- 4Grade every failure — by code or by judge
- 5Default to pass/fail, not a 1–5 scale
- 6Validate the judge against a human before you trust it
- 7Turn every fixed bug into a regression test
- 8For agents, grade the outcome — not the exact steps
2Techniques
Learn
Look first
Error analysis on real failures
Read real traces before you build any metric The fastest way to a good eval is not a dashboard — it is sitting down with real outputs and writing down what went wrong, in your own words.
✕Instead of
Skim a handful of outputs, decide they "look good," and ship.
✓ Try this 💬 AI chat
Collect real traces (a trace = the full record of one request → response). For each, write an open-ended note on the FIRST thing that went wrong. Read until new traces stop revealing new problems — aim for at least ~100.
Why it works: Hamel Husain calls error analysis "the most important activity in evals." You keep reading "until you reach theoretical saturation, meaning new traces do not seem to reveal new failure modes" — "you should aim to review at least 100 traces." The notes you write are the raw material every later metric is built from.
Group the notes into a failure taxonomy Loose notes become a tool the moment you cluster them: a short, named list of the distinct ways your system actually fails.
✕Instead of
Keep a vague mental list of "stuff that goes wrong sometimes."
✓ Try this 💬 AI chat
Sort your open-ended notes into a handful of named failure modes (e.g. "ignored a constraint", "wrong citation", "made up a number"). Count how often each one happens.
Why it works: This is open coding → axial coding from qualitative research: write free-form notes, then categorise them into a taxonomy. The counts tell you which failure is worth an eval first — you fix the common, real failure, not the one that is easiest to imagine.
Only measure failures you have actually seen A metric is worth building when it tracks a real, recurring failure from your data — not because a library ships it.
✕Instead of
Bolt on a generic "helpfulness", "coherence" or "toxicity" score from an eval library and call it done.
✓ Try this 💬 AI chat
Turn each failure mode you observed into one specific, named check. Skip the prefab scores unless a real failure maps onto them.
Why it works: Generic metrics "measure abstract qualities that may not matter for your use case" — as Hamel puts it, "good scores on them don't mean your system works." A failure-driven metric is grounded in behaviour you can point to, so improving it actually improves the product.
Grade it automatically
Code · LLM-judge · human
Grade every failure — by code or by judge Two ways to grade, pick the cheapest that works. Many failures are checkable with plain code: exact match, a regex, valid JSON, a number in range. When "right" needs reading and interpreting — tone, faithfulness to a source, did-it-answer — have a model grade it against a clear rubric instead.
✕Instead of
Spin up an LLM-as-judge to check whether the output is valid JSON or contains a required field.
✓ Try this 💬 AI chat
Write a code assertion: parse the JSON, assert the field exists, assert the total is within range. Reserve the LLM judge for the calls that genuinely need judgement.
✕Instead of
Ask the judge: "Rate this answer from 1 to 10." (And use the same model that wrote the answer.)
✓ Try this 💬 AI chat
Give the judge a specific rubric, have it reason in <thinking> tags then output only pass/fail, and grade with a DIFFERENT model than the one that generated the answer.
Why it works: Code-based grading is "fastest and most reliable, extremely scalable" (Anthropic). Hamel's rule of thumb: "if you can catch an error with a simple assertion or regex check, the cost is minimal and probably worth it." Spend the expensive judge only where code cannot reach. For the judgement calls, LLM grading is "fast and flexible, scalable and suitable for complex judgement" (Anthropic), which also advises you to "ask the LLM to think first… then discard the reasoning" and that it is "best practice to use a different model to evaluate than the model used to generate." A vague 1–10 prompt produces noise; a rubric produces a signal.
Default to pass/fail, not a 1–5 scale A binary verdict forces you to define what actually matters. Numeric scales feel precise but smuggle in disagreement.
✕Instead of
Score every output 1–5 for "quality" and track the average.
✓ Try this 💬 AI chat
Define one clear pass/fail question per failure mode ("Does the answer cite a real source from the provided context? yes/no"). Use a number only on the rare axis where the gradient genuinely carries information.
Why it works: Hamel & Shreya: "Binary evaluations force clearer thinking and more consistent labeling" — "people don't know what to do with a 3 or 4." (Worth knowing the tension: Anthropic's own docs use Likert/ordinal scales for nuanced axes like tone — so reach for a scale only when a real gradient earns it, and define each point.)
Validate & lock in
Trust the judge, then regress
Validate the judge against a human before you trust it An LLM judge is just another model that can be wrong. Until you have checked it against human labels, its "94% pass" means nothing.
✕Instead of
Ship the LLM judge and report its scores as truth.
✓ Try this 💬 AI chat
Have ONE domain expert label a held-out set pass/fail. Compare the judge to those labels and measure precision & recall (not just raw agreement) until they line up; feed the expert's critiques back in as few-shot examples.
Why it works: Hamel's judge recipe starts by finding "THE principal domain expert" and warns that "using raw agreement is generally not recommended… when classes are imbalanced — instead, measure precision and recall separately." This is the "who validates the validators" problem: the grader needs a human check, or you have only moved the trust problem, not solved it.
Turn every fixed bug into a regression test An eval you run once is a snapshot. An eval in CI is a ratchet — it stops you silently re-breaking what you already fixed.
✕Instead of
Eval once before launch, then change the prompt freely and hope.
✓ Try this 💬 AI chat
Add each discovered failure as a case in a saved eval set that runs on every change. Keep two suites: regression evals you expect to stay green, and capability evals that start hard.
Why it works: Anthropic: "Capability or 'quality' evals… should start at a low pass rate," while "regression evals… should have a nearly 100% pass rate." Pinning fixed failures into a green-on-every-run suite is what lets you iterate on prompts and models without quietly regressing.
For agents, grade the outcome — not the exact steps A multi-step agent can reach a correct result by many paths. Grading the path punishes creativity and breaks constantly.
✕Instead of
Assert the agent called tool A, then tool B, then tool C in that exact order.
✓ Try this 💬 AI chat
Grade the end state — did it produce the right answer / file / change? Start with 20–50 tasks drawn from real failures, and READ the transcripts to check your grader is fair.
Why it works: Anthropic: "it's often better to grade what the agent produced, not the path it took," "20-50 simple tasks drawn from real failures is a great start," and "you won't know if your graders are working well unless you read the transcripts." (A 0% pass@100 "is most often a signal of a broken task, not an incapable agent.")
3Lessons 5
3.1 Block drops in extraction quality with a regression suite
A set of automated tests built from previously observed mis‑read invoices that must maintain at least 98% accuracy after any code change.
Fail loudly if extraction performance falls below the defined threshold
- Select the invoices flagged as errors and add them to a test dataset
- Write a test script that runs the extractor on each invoice and checks exact‑match against the known correct fields
- Add an assertion that the overall pass rate must be ≥ 98% and configure your CI pipeline to treat failures as build breaks
- Run the suite before and after making a change to verify it still passes
- You'll see CI builds succeed when accuracy stays above 98% and abort with an error message listing any invoices that regressed
- Takeaway Embedding concrete quality gates in your development workflow prevents accidental re‑introduction of bugs
- Check What outcome does your CI pipeline produce when the extraction pass rate falls below the 98% threshold?
3.2 Create a repeatable evaluation dataset for an extraction use‑case
An evaluation dataset is a collection of input prompts paired with the correct expected output that you can run repeatedly to measure model performance.
You will have a JSON file containing at least five inputs and their ground‑truth extraction results ready for automated testing.
- Open a text editor and create a new file named
extraction_eval.json. - For each test case, write an object with two keys:
input(the raw text to process) andexpected(the exact field values you expect the model to extract). - Add at least five such objects, covering typical cases and one edge‑case where extraction often fails.
- Save the file and validate that it is valid JSON (e.g., by running
python -m json.tool extraction_eval.json).
- You'll see A well‑formed
extraction_eval.jsonfile containing a list of input/expected pairs, ready to be consumed by an evaluation framework. - Takeaway Defining clear, repeatable test cases is the foundation for trustworthy AI evals and later regression testing.
3.3 Pick the cheapest model that meets your extraction quality bar
An evaluation that runs identical field‑level accuracy tests on multiple model versions, then compares pass rates to cost to identify the lowest‑cost option that still satisfies a target threshold.
Choose a model version that balances cost and performance based on actual evaluation data
- Identify three candidate models (e.g., small, medium, large) and note their per‑token pricing in a table
- Select each model in the Model selector and run the field‑level extraction eval from Lesson 1 using the same gold set via the Run evaluation button
- Record each model’s overall accuracy and compute its pass rate against the predefined quality bar (e.g., ≥ 95%) in the Results pane
- Compare the models’ costs versus their pass rates in the Comparison view and choose the cheapest model that meets or exceeds the quality bar
- You'll see A concise report listing each model, its cost per 1 k tokens, its accuracy percentage, and a clear recommendation of the lowest‑cost model that satisfies the target
- Takeaway Data‑driven model selection avoids overpaying for unnecessary capacity while maintaining required performance
- Check How do you identify the cheapest model that still satisfies the predefined quality bar of at least 95% accuracy?
3.4 Add the DeepEval script to a CircleCI pipeline using the evals orb
CircleCI’s evals orb simplifies defining evaluation jobs, running them in CI, and asserting that results meet thresholds with CEL expressions.
Your repository will run the extraction eval on every push and fail the build if accuracy drops below a set threshold.
- Create a
.circleci/config.ymlfile if one does not exist. - Add the
orbs: { evals: circleci/evals@x.y }line (use the orb name as described on the CircleCI docs page). - Define a job that installs dependencies, runs
python run_extraction_eval.py > results.json, and then callsevals/run:withresult_file: results.json. - Add a test case in the same job using CEL, e.g.,
accuracy > 0.90, to make the pipeline fail when the threshold is not met.
- You'll see When you push a commit, CircleCI executes the eval job, prints the G‑Eval scores, and either passes or fails the workflow based on the CEL assertion.
- Takeaway Automating evals in CI ensures every code change is validated against your quality bar before it reaches production.
3.5 Create a regression suite that locks in fixes for discovered extraction failures
A regression suite is a collection of eval cases that capture known failure modes so future changes cannot re‑introduce them.
You will extend extraction_eval.json with failing examples, tag them, and configure the CI job to run only these regression tests after a fix.
- Run the DeepEval script on the current codebase and note any cases where the G‑Eval score is below 1.0.
- Copy those failing objects into a new file
extraction_regression.jsonand add a field ` - type": "regression"` to each entry.
- Update
run_extraction_eval.pyto accept an optional argument--suitethat loads either the full dataset or the regression suite. - Modify the CircleCI config to run a second job after a fix merge that executes
python run_extraction_eval.py --suite extraction_regression.jsonand asserts all scores equal 1.0.
- You'll see A separate CI job that only passes when previously failing cases are fully corrected, providing a safety net against regressions.
- Takeaway Regression suites turn discovered bugs into permanent checks, turning ad‑hoc fixes into repeatable quality guarantees.
4You’ll know it worked 27 checkable outcomes in this chapter
- ✓A table of per-field accuracies and a list of papers with extraction errors
- ✓A confusion matrix is displayed with precision and recall per class, highlighting the class with lowest recall
- ✓Build fails when pass rate drops below 98%
- ✓Confusion matrix displayed with per-class recall for urgent and security tickets
- ✓Higher score bands show higher quarterly close rates
- ✓Resolution rate displayed and list of unresolved conversations shown
- ✓A list of all unsupported claims is produced, ready for citation or removal
- ✓The output lists pass rates and costs for each model, highlighting the cheapest one that meets the quality threshold
27 outcomes in all — one per recipe below.
5FAQ, Tips & How-to 35
one problem, one solution, one action
Research & data tools4
▸How-to Scientist +1
You catch hallucinated citations automatically, instead of discovering them when a reader does.
▸How-to Scientist +1
You see where the classifier is weak (often the rare-but-important class) instead of trusting a flattering average.
▸How-to Physician
A measured accuracy and hallucination rate before the assistant ever drafts a summary a physician signs off on — and a concrete list of exactly which claim types it fabricates most, so review effort focuses there. No output reaches a patient record without a clinician confirming it first.
Knowledge & docs3
▸How-to Scientist +1
A ranked list of how your summariser actually fails — so you build the eval that matters first, not the one that is easiest to imagine.
▸How-to HR / People +1
You learn how often the bot invents policy, and on which topics, before employees rely on it.
▸How-to Support +1
You catch answers that are confidently wrong before a customer acts on them.
Dashboards & analytics2
▸How-to Finance +1
Confidence that auto-categorisation is safe to trust for the books — with the weak categories named.
▸How-to Founder +1
Which AI model is the cheapest that still meets my quality requirements?
A model choice backed by data — the cheapest model that still passes, not the priciest by default.
Internal tools & ops5
▸How-to Finance +1
You only roll out the judge once it actually matches a human — not on the assumption that it does.
▸How-to HR / People +1
Hard evidence of whether the screener is consistent before it ever touches a real candidate.
▸How-to Operations +1
You know the router won't quietly misroute the 2% of tickets that actually matter most.
▸How-to Operations +1
Check if the operations agent hits its goal no matter the steps
A robust eval that rewards the agent for getting the job done, not for following one rigid script.
▸How-to Founder +1
Not sure a new feature is ready
A go/no-go gate that catches embarrassing failures before customers do, not after.
Forms, surveys & feedback2
▸How-to Finance +1
Invoices get mis‑read after changes
You can improve the extractor freely, knowing the build fails loudly if an old bug comes back.
▸How-to Operations +1
Records missing required data get flagged early
Cheap, fast checks catch most errors; the expensive judge runs only where it earns its keep.
Content & marketing5
▸How-to HR / People +1
Off-brand or non-inclusive JDs are caught automatically before a recruiter posts them.
▸How-to Sales +1
You ship a drafting tool that won't embarrass the team with a hallucinated detail or a missing ask.
▸How-to Creator +1
Unsure if generated copy stays on brand
Content that stays recognisably yours, even as volume scales up.
▸How-to Creator +1
You publish with fewer confident-but-wrong claims, and a record of which ones needed a source.
▸How-to Creator +1
Need to check meta descriptions for length, keywords and cut‑offs
Hundreds of meta-descriptions checked in seconds, with zero LLM cost, before they go live.
CRM & sales2
▸How-to Sales +1
Evidence of whether the score is predictive before the team reorders its day around it.
▸How-to Sales +1
CRM notes you can trust, because every line is grounded in what was actually said.
Customer & client portals3
▸How-to Support +1
A real resolution rate to track over time — the metric that actually predicts customer satisfaction.
▸How-to Support +1
Unsure when the bot should hand off
You tune the escalation threshold on evidence, balancing customer pain against agent load.
▸How-to Small biz +1
A chatbot that answers your top questions correctly from day one, instead of learning on real customers.
Trackers1
▸How-to Small biz +1
Manually spot‑checking prompts after changes
The checks you do informally become a one-click test you can run after every change.
▸How-to Everyone
When you only skim a few outputs
The fastest way to a good eval is not a dashboard — it is sitting down with real outputs and writing down what went wrong, in your own words. Hamel Husain calls error analysis "the most important activity in evals." You keep reading "until you reach theoretical saturation, meaning new traces do not seem to reveal new failure modes" — "you should aim to review at least 100 traces." The notes you write are the raw material every later metric is built from.
▸How-to Everyone
You only want to track real, recurring errors
A metric is worth building when it tracks a real, recurring failure from your data — not because a library ships it. Generic metrics "measure abstract qualities that may not matter for your use case" — as Hamel puts it, "good scores on them don't mean your system works." A failure-driven metric is grounded in behaviour you can point to, so improving it actually improves the product.
▸How-to Everyone
I have a vague list of failures
Loose notes become a tool the moment you cluster them: a short, named list of the distinct ways your system actually fails. This is open coding → axial coding from qualitative research: write free-form notes, then categorise them into a taxonomy. The counts tell you which failure is worth an eval first — you fix the common, real failure, not the one that is easiest to imagine.
▸How-to Everyone
When you rate output on a 1‑5 scale
A binary verdict forces you to define what actually matters. Numeric scales feel precise but smuggle in disagreement. Hamel & Shreya: "Binary evaluations force clearer thinking and more consistent labeling" — "people don't know what to do with a 3 or 4." (Worth knowing the tension: Anthropic's own docs use Likert/ordinal scales for nuanced axes like tone — so reach for a scale only when a real gradient earns it, and define each point.)
▸How-to Everyone
Simple code checks, complex rubric grading by an LLM
Two ways to grade, pick the cheapest that works. Many failures are checkable with plain code: exact match, a regex, valid JSON, a number in range. When "right" needs reading and interpreting — tone, faithfulness to a source, did-it-answer — have a model grade it against a clear rubric instead. Code-based grading is "fastest and most reliable, extremely scalable" (Anthropic). Hamel's rule of thumb: "if you can catch an error with a simple assertion or regex check, the cost is minimal and probably worth it." Spend the expensive judge only where code cannot reach. For the judgement calls, LLM grading is "fast and flexible, scalable and suitable for complex judgement" (Anthropic), which also advises you to "ask the LLM to think first… then discard the reasoning" and that it is "best practice to use a different model to evaluate than the model used to generate." A vague 1–10 prompt produces noise; a rubric produces a signal.
▸How-to Everyone
Can’t trust an AI grader by itself
An LLM judge is just another model that can be wrong. Until you have checked it against human labels, its "94% pass" means nothing. Hamel's judge recipe starts by finding "THE principal domain expert" and warns that "using raw agreement is generally not recommended… when classes are imbalanced — instead, measure precision and recall separately." This is the "who validates the validators" problem: the grader needs a human check, or you have only moved the trust problem, not solved it.
▸How-to Everyone
Want to judge an agent by its answer, not its step‑by‑step path
A multi-step agent can reach a correct result by many paths. Grading the path punishes creativity and breaks constantly. Anthropic: "it's often better to grade what the agent produced, not the path it took," "20-50 simple tasks drawn from real failures is a great start," and "you won't know if your graders are working well unless you read the transcripts." (A 0% pass@100 "is most often a signal of a broken task, not an incapable agent.")
▸How-to Everyone
Every bug you fix keeps breaking later
An eval you run once is a snapshot. An eval in CI is a ratchet — it stops you silently re-breaking what you already fixed. Anthropic: "Capability or 'quality' evals… should start at a low pass rate," while "regression evals… should have a nearly 100% pass rate." Pinning fixed failures into a green-on-every-run suite is what lets you iterate on prompts and models without quietly regressing.
The same set on /recipes, filtered by tool and role.
6Videos 2
Why AI evals are the hottest new skill for product builders
The creators of the most popular evals course on why evals matter and why error analysis beats generic metrics. Start here for the mindset.
How to Build AI Evals in 2026 (Step-by-Step, No Hype)
A hands-on walk through the full loop on real production data — error analysis, an LLM judge, and validating the judge against human labels.
7FAQ 6
What is an eval, and why isn't a vibe-check enough?
An eval is a repeatable test of whether your AI does what you need — a measurement you can re-run after every change. A vibe-check (reading a few outputs and deciding they look fine) doesn't scale, isn't comparable across versions, and hides the failures you didn't happen to sample. The highest-value first step is error analysis — actually reading real traces and writing down what went wrong — which Hamel Husain calls "the most important activity in evals."
Code-based, LLM-as-judge, or human grading — which should I use?
Reach for the cheapest grader that works. Code-based grading (exact match, regex, schema checks) is "fastest and most reliable, extremely scalable" — use it whenever the check is an if/else. Use an LLM-as-judge for calls that need interpretation (tone, faithfulness, did-it-answer); it is "fast and flexible… suitable for complex judgement," but test it first. Human grading is the highest quality and the calibration gold standard, but slow and expensive — "avoid if possible," and use it to validate the others.
Should my judge score 1–5 or just pass/fail?
Default to binary pass/fail. As Hamel & Shreya put it, "binary evaluations force clearer thinking and more consistent labeling" — with a 1–5 scale "people don't know what to do with a 3 or 4," and adjacent points mean different things to different annotators. Use a numeric scale only on the rare axis where the gradient genuinely carries information, and define each point. (Anthropic's docs do use Likert/ordinal scales for nuanced axes like tone — so the rule is "binary by default, numeric where it earns its keep," not "never use a scale.")
How do I know my LLM judge is actually trustworthy?
Validate it against a human. Have one principal domain expert label a held-out set pass/fail, then compare the judge to those labels. Measure precision and recall, not just raw agreement — "using raw agreement is generally not recommended… when classes are imbalanced." Iterate the judge prompt (feeding the expert's critiques in as few-shot examples) until it lines up. An unvalidated judge just moves the trust problem; it doesn't solve it.
How do I evaluate a multi-step AI agent?
Grade the outcome, not the path: "it's often better to grade what the agent produced, not the path it took," because asserting an exact sequence of tool calls is rigid and punishes valid alternatives. Start small — "20-50 simple tasks drawn from real failures is a great start" — and read the transcripts, because "you won't know if your graders are working well unless you read the transcripts." A 0% pass rate usually means a broken task, not an incapable agent.
How many test cases do I need, and do I have to write code?
Fewer than you think to start, and not necessarily. "Prioritize volume over quality" — more cases with decent automated grading beats a handful of hand-graded ones — but you can begin with 20–50 real failures and grow. You don't need to code for the first two steps (read traces, label pass/fail — a spreadsheet works); code-based graders and CI regression are the power-user rung. DeepLearning.AI's "Evaluating AI Agents" (with Arize) is a good hands-on next step.
8Glossary 17 terms
Show the 17 terms
Concepts
Eval- A repeatable test of whether an AI system does what you need — re-run after every change to see if it got better or worse.
Trace- The complete record of one run: the input, every step/tool call, and the final output. What you read during error analysis.
Error analysis- Reading real traces and writing open-ended notes on the first failure in each — the most valuable first step in evals.
Failure mode- A distinct, named way the system goes wrong (e.g. "ignored a constraint", "wrong citation"), found by grouping your notes.
Failure taxonomy- The short, named list of failure modes you build by categorising error-analysis notes (open coding → axial coding).
Theoretical saturation- The point where reading more traces stops revealing new failure modes — a sign you have seen enough to start measuring.
Ground truth- The known-correct answer (or human pass/fail label) you grade an output against. Also called the golden or reference answer.
Held-out set- Labelled examples kept aside to check a grader or model honestly — not the ones used to build or tune it.
Methods
Code-based grader- A plain-code check (exact match, regex, JSON-valid, number-in-range). Fastest and most reliable — use whenever the check is an if/else.
LLM-as-judge- Using a model to grade another model's output against a rubric. For judgement calls; reason-then-discard, output binary, grade with a different model.
Binary vs Likert- Pass/fail (binary) vs a 1–5 scale (Likert). Default to binary — it forces clear definitions; use a scale only where a real gradient earns it.
Domain expert- The one person whose pass/fail judgement defines "correct" for your product. Their labels validate the judge; their critiques become few-shot examples.
Precision & recall- How often the judge's "pass" is right (precision) and how many true passes it catches (recall). Preferred over raw agreement when classes are imbalanced.
Synthetic data- Test cases generated from structured dimensions (feature × scenario × persona) rather than asked for in bulk — better coverage, less repetition.
Regression eval- A saved suite of previously-fixed failures, run on every change; expected to stay near 100% so you don't silently re-break things.
Capability eval- An eval of what the system can do well; expected to start at a low pass rate and climb as you improve it.
pass@k- The chance of at least one success in k tries. A 0% pass@100 usually means a broken task, not an incapable model.
9See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.