▸How-to Everyone
Not sure what correct AI output looks like
Specifying what a correct answer looks like first prevents reliance on lucky outputs
Track 3 · Beginner
3.3
Test-Driven Prompt Engineering
Stop guessing whether a prompt got better — prove it
1Overview
Writing your `input → expected` examples before you write the prompt, then editing the prompt until every example passes — the red → green → refactor loop borrowed from software's test-driven development.
A discipline, not a tool: borrow software's Test-Driven Development for prompting. Write a few input → expected examples first, then let the failing cases drive how you write and refine the prompt — the red → green → refactor loop. It turns "this answer looks good" into a checkable spec, so a prompt edit, a model swap, or a silent model update cannot break something quietly. → Unlike Prompt Engineering (which techniques to reach for), this chapter is the workflow that makes any of them dependable. → For how to actually grade the outputs — code checks, an LLM-as-judge, human review — see Evals & testing; TDPE is the loop, Evals is the machinery. Tool-agnostic; promptfoo is the running example.
1.1Before you start
1.2After this chapter you can
→Write a small "golden set" of input → expected examples before you touch the prompt
→Let a failing example drive each prompt edit — red → green → refactor
→Re-run the set on every prompt change, model swap or model update to catch regressions
→Know where TDPE stops and Evals & testing begins — cross-link, don't duplicate
1.3When to reach for it
The moment you catch yourself tuning a prompt by eye on one lucky output — or a prompt that worked last month quietly breaks after a model update.
1.4Key parts
A small example file (5-10 real input/output pairs, edge cases included), the red → green → refactor loop, and re-running the file on every prompt edit or model swap.
1.5How it differs from Evals
TDPE is the habit of writing examples first and re-running them — it does not teach you how to grade an answer. For code checks, LLM-as-judge, or a full regression suite, see the Evals & testing chapter.
2Lessons 6
2.1 Validate prompts with test cases
A test case is an input → expected output pair written before the prompt exists, turning “does this work” into a checkable condition.
Prove a prompt works reliably by defining and testing expected outputs before writing it
Try
Add a test case: Input – “Student asks about deadline”; Expected output – “The deadline is next Friday.” Then run the test suite.Enter this text in the Test Cases section of the test-driven-prompt-engineering interface and press Run Tests. Look for a red (failed) result before you start editing the prompt.
- Install promptfoo on your machine using the recommended installer
- Create a test file containing concrete
input → expected outputpairs for the task you want to solve - Write an initial prompt that attempts to produce the expected outputs
- Execute promptfoo eval on each input and observe which cases fail (red)
- Edit the prompt minimally until all examples produce the correct result (green)
- You'll see A list of input‑output examples that the prompt passes, confirming it is ready
- Takeaway One good answer does not prove reliability; define what “right” looks like first and test against it
- Check Why does a prompt that produced one good answer still count as unproven?
- Cost No tool cost here — this lesson is a five-minute mindset reset. The real cost/time trade-off shows up once you start writing examples in the next lesson.
2.2 Create a concrete example specification
The example file is a short table or plain‑text list of real inputs beside their correct outputs, grown by adding rows whenever the task fails in a new way.
Do this first Validate prompts with test cases
Produce a written, checkable spec of inputs and expected outputs before drafting any prompt
Try
Create a plain‑text spec file named `spec.txt` containing five real input→expected pairs for the task “classify student emails as urgent or not”. Use actual email excerpts (you can copy them from your inbox) and write, in one sentence each, what the correct classification should be. Include at least one edge case you’ve seen mis‑classified before.Paste the prompt into the New Spec field on the test-driven-prompt-engineering dashboard and hit Generate. Verify that each line follows the input → expected format and that an edge case is present.
- Write 5‑10 real input → expected pairs drawn from actual data such as student emails or research abstracts
- Include edge cases and previously failed examples to capture difficult scenarios
- Add a new example each time you discover another way the task can go wrong, expanding the spec over time
- You'll see A short list – a table or plain‑text file – showing real inputs alongside the correct output for each one
- Takeaway Your example file becomes the living specification that guides prompt development and guards against past failures
- Check Why does a previously failed example earn a permanent place in the example file?
- Cost Fifteen to thirty minutes for a first batch of 5-10 examples. It feels slow compared to just writing a prompt, but it's the only step that turns tuning-by-eye into something you can actually check.
2.3 Drive prompt edits using a failing example
The red case is the first example your draft prompt fails, and it is the only thing the next edit is allowed to be about.
Do this first Create a concrete example specification
Find the first example your draft prompt fails and apply the smallest edit that makes it succeed
Try
Run the current draft prompt on every example in the suite and output a pass/fail table.Paste this into the Prompt Runner input box of the test-driven-prompt-engineering tool and hit Run. Watch the results grid and note the first row that turns red – that’s the target example to fix.
- Run your draft prompt on each example using promptfoo eval and mark each as pass or fail
- Identify the first failing example – the red case – as the edit target
- Edit the prompt with the smallest change that turns that case green
- Re‑run all examples to ensure no other rows have turned red
- You'll see A pass/fail grid where one red row turns green after a targeted edit while all other rows remain green
- Takeaway Target the specific failure with the minimal defensible change and verify every example again — never optimise a whole prompt from a single happy output
- Check After the smallest edit turns the red row green, what must you do before accepting it?
- Cost A few minutes per red example if the fix is a small addition (an example, a role); longer if the failure reveals the task itself was underspecified and needs rethinking.
2.4 Tighten a passing prompt without breaking it
Refactoring a prompt is removing a repeated instruction or an over‑long example while every test stays green — the shorter wording is a hypothesis, not an improvement, until the suite says so.
Do this first Drive prompt edits using a failing example
Refactor the prompt for clarity and lower cost while all examples stay green
Try
Refactor the prompt in prompts.txt to make it clearer and shorter while ensuring every example still passes the test suite.Paste this command into the Prompt Editor screen of the test-driven-prompt-engineering tool, then click Run Tests. Watch for any examples turning red – that indicates a regression you must fix before proceeding.
- Identify repeated instructions or overly long examples while all tests are green and plan a single edit
- Remove or rewrite the chosen redundant phrase, treating the shorter version as a hypothesis
- Re‑run the test suite immediately after the edit to confirm no example has failed
- You'll see A shorter, cleaner prompt that still passes every example in your file
- Takeaway Refactor the prompt the same way you would refactor code under a green test suite: tighten it, then re‑run, never the reverse
- Check What has to be true of the test suite before you shorten a prompt, and what tells you the edit held?
- Cost Shorter prompts cost less per call and read easier six months from now, but every trim needs a re-run, so refactoring is slow, deliberate work — budget real time for it, not five minutes.
2.5 Re‑run examples after every change
The example file lives beside the prompt so it travels with it, and it is run again after every prompt edit, every model swap, and every silent change at the provider.
Do this first Tighten a passing prompt without breaking it
Make the example file a permanent companion to the prompt and execute it whenever you edit the prompt, switch models or the provider updates
Try
test-driven-prompt-engineering run example_prompts.csvPaste this command into the Terminal of the test-driven-prompt-engineering UI after saving your example file next to the prompt. Watch the output table for any red ✗ entries indicating a regression before accepting the changes.
- Store the example file beside the prompt so it travels with the prompt code
- Edit the prompt or change the model, then re‑run the example file manually using promptfoo eval
- Inspect the resulting pass/fail grid before accepting the new version
- You'll see A pass/fail grid appears after each run, showing whether the new version still meets the expected output
- Takeaway An example file only protects you when you re‑run it on every prompt edit model swap or silent provider update
- Check Which changes — not just your own edits — should send you back to the pass/fail grid?
- Cost A manual re-run of 10 examples takes a few minutes each time — trivial compared to the cost of a regression reaching real users; a tool like promptfoo trades a bit of setup time for near-zero re-run cost afterward.
2.6 Run a full test‑driven prompt workflow
The red‑green‑refactor loop run end to end on a real task: write examples, edit until green, tighten, then execute the same unchanged file against a different model.
Do this first Re‑run examples after every change
Complete a real non‑code task using the red‑green‑refactor loop and verify it on a different model
Try
In the test-driven-prompt-engineering UI, click **New Prompt**, paste:
Classify each student email into one of: question, deadline, complaint, other. If multiple categories apply, list them separated by commas.
Then press **Run** on the six example emails you prepared.Enter this in the Prompt Editor screen and hit Run. Observe the pass/fail grid – all rows should be green before you switch models.
- Select a relatable task and write six examples, including at least one edge case
- Draft an initial prompt and run it against all six examples to identify red cases using promptfoo eval
- Edit the prompt minimally to turn the first failing example green, repeating until all are green
- Tighten the prompt for clarity and re‑run the examples to ensure nothing breaks
- Swap to a different model or provider and execute the same unchanged file with promptfoo eval
- Observe which examples turn red, indicating regression across models
- You'll see A pass/fail grid that stays green after refactoring and highlights failures when the model is swapped
- Takeaway Examples drive edits, green confirms safety for refactoring, and re‑running on every change builds prompt trustworthiness
- Check What does swapping the model and re‑running the same example file reveal about the prompt?
- Cost Budget the full 20 minutes for a first end-to-end pass on a new task — writing 6 honest examples is the part that takes longest, everything after it is fast by comparison.
3You’ll know it worked 38 checkable outcomes in this chapter
- ✓The grid shows at least one red row indicating a failing example
- ✓A list of pass/fail test results appears in the console output or is written to a result file
- ✓All unit tests pass after each step
- ✓The model returns output matching the requested format, style, and audience
- ✓Model retains all required information in its response without truncation
- ✓Running `npm run dev` starts a local server that serves the app without errors
- ✓Run the full test suite; all tests return green (pass) status
- ✓Running the test file shows all tests passing, confirming the function meets the described criteria
38 outcomes in all — one per recipe below.
4FAQ, Tips & How-to 59
one problem, one solution, one action
▸How-to Everyone
Want to see if a prompt meets its goal
Concrete example pairs let you objectively evaluate whether a prompt meets its goals
▸How-to Everyone
My prompt keeps failing
Applying the TDD red→green→refactor cycle to prompts yields reliable, maintainable wording
!Tip Everyone
Defendability check — can you justify a prompt after one failure?
If you cannot defend a prompt to someone who only saw it fail once, the prompt is still tuned by intuition rather than evidence
▸How-to Everyone
Your prompt keeps missing tricky cases
Adding examples that previously broke the task forces the prompt to handle the hardest situations
▸How-to Everyone
I only have a few examples
Beginning with a few examples keeps the effort manageable, and you can continuously add new cases as they appear
▸How-to Everyone
Need a clear prompt specification you can verify
A written spec lets you verify prompt quality objectively rather than by eye
▸How-to Everyone
Unsure which prompt examples fail
You can see which examples your draft prompt fails on before making changes
▸How-to Everyone
You concentrate your fixing effort on the specific example that fails first
▸How-to Everyone
A failing prompt needs a tiny tweak
A tiny, defensible tweak fixes the failure without over-hauling the prompt
▸How-to Everyone
Ensures the fix didn't break any previously passing cases
▸How-to Everyone
Confirms the change is truly required, not a guess
▸How-to Everyone
You only start trimming a prompt after every test example is passing, ensuring you don't break functionality
▸How-to Everyone
Examples get lost in another folder
Storing the test examples alongside the prompt ensures you always remember to run them
▸How-to Everyone
Manually grading examples in a spreadsheet
You can begin testing with a simple spreadsheet and manual grading before automating
▸How-to Everyone
Manually rerunning examples for each prompt and model
Using promptfoo lets you rerun all examples with a single command, removing manual effort
▸How-to Everyone
Need clear examples to shape a prompt
Starting with a set of concrete examples lets you define correct behavior before writing any prompt
▸How-to Everyone
My prompt keeps failing tests
Running the prompt against examples and fixing failures (red) until all pass (green) produces a reliable prompt
▸How-to Everyone
Making the smallest possible change (adding a role, example, or delimiter) resolves failures without over-engineering
▸How-to Everyone
Prompt is too long after passing all tests
After achieving all-green, you can streamline the prompt and verify stability by re-running the test suite
▸How-to Everyone
Running the same test suite on a different model quickly reveals silent regressions that would affect real users
?FAQ Everyone
Why does one correct answer not prove my prompt works reliably?
A single successful output is just luck; it doesn’t show the prompt will handle variations in inputs. Reliability means the prompt consistently gives right answers across many situations, not just once.
?FAQ Everyone
How do I know what a correct AI response should look like before writing the prompt?
First write down the criteria for a “right” answer—specific wording, format, or content you expect. This definition guides you to craft a prompt that aims for those exact results instead of hoping for a lucky match.
?FAQ Everyone
What are input → expected output pairs and why should I create them?
They are concrete examples where you list an actual user input and the exact answer you want the AI to produce. Having these pairs lets you objectively test whether your prompt meets its goal.
?FAQ Everyone
Can you explain the red‑green‑refactor cycle for fixing prompts?
Start with a failing example (red), make the smallest change that makes it pass (green), then once all examples pass, tidy up the wording without breaking anything (refactor). This disciplined loop keeps edits minimal and the prompt stable.
?FAQ Everyone
How do I handle edge cases that previously broke my prompt?
Add those problematic scenarios as extra input → expected output pairs in your specification. By testing against these hard cases you force the prompt to handle them correctly.
!Tip Everyone
Smart Zone vs Dumb Zone — keep LLM tasks within optimal context
LLMs perform best when the number of tokens in a single request stays below roughly 100k, which Matt calls the "smart zone". Beyond that the attention graph grows quadratically and the model's output degrades into the "dumb zone".
!Tip Everyone
Multi‑Phase / Vertical Slice Planning — avoid the dumb zone while delivering features
Break a large feature into small, end‑to‑end vertical slices that cross all layers (schema, API, UI). Each slice can be completed and tested before moving to the next, keeping every request inside the smart zone.
!Tip Everyone
Grill Me Skill — align with LLM by iteratively questioning requirements
The "grill me" skill prompts the model to ask you focused clarification questions, building a shared design concept before any code is written. This reduces mis‑alignment and keeps the conversation token‑efficient.
!Tip Everyone
Compacting Context — reset or compress conversation to stay in the smart zone
After a long interaction, you can either clear the context entirely or use a "compact" operation that summarizes prior exchanges into a short summary, freeing token space while preserving essential information.
▸How-to Everyone
Add a new feature without breaking existing code
TDD consists of five steps: write a failing test, verify it fails, write minimal code to pass, ensure all tests pass, then refactor. This forces you to clarify requirements before coding and keeps the codebase stable.
▸How-to Everyone
Tests interfere with each other's data
Using setUp creates a fresh fixture before each test method, preventing state leakage between tests. tearDown can clean resources after each test.
!Tip Everyone
assertIsInstance & assertAlmostEqual — robust numeric assertions
assertIsInstance checks the type of a result (e.g., float), while assertAlmostEqual handles floating‑point rounding errors by allowing a tolerance.
!Tip Everyone
Avoid Shared Test Data — keep tests independent
Reusing a single object across multiple test methods creates hidden dependencies; each test should create its own data or use setUp to get a fresh copy.
!Tip Everyone
Fixed Expected Values — write meaningful assertions
Comparing a function’s output to a hard‑coded expected value validates behavior, whereas comparing to another implementation merely mirrors code and can miss bugs.
▸How-to Everyone
Want consistent AI answers every time
A well‑crafted prompt combines context, task, persona, format, examples, and tone. Specifying each element guides the LLM to produce targeted, formatted output.
▸How-to Everyone
Prompt is vague and gets rambling replies
A weak prompt is vague (e.g., "Tell me about AI"), while a strong prompt adds audience, length, and format constraints, leading to concise, relevant answers.
▸How-to Everyone
Want a blog post but don’t know the structure
Dividing a large request into sequential sub‑tasks lets the model brainstorm, select, and then expand, giving you control over each stage.
▸How-to Everyone
Need content that sounds like a journalist
Assigning a persona (e.g., journalist, nutritionist) influences vocabulary and perspective, producing content that fits the intended voice.
▸How-to Everyone
My prompts are too long and costly
LLMs process text as tokens (≈4 characters each). Shorter, token‑efficient prompts reduce cost and improve model focus.
▸How-to Everyone
Each model has a maximum token window (e.g., GPT‑3.5 ≈8k, GPT‑4 ≈32k). Exceeding it causes earlier content to be dropped, leading to incomplete answers.
▸How-to Everyone
Prompt keeps missing key info
After receiving an output, analyze gaps and adjust the prompt (add constraints, clarify intent). Re‑prompting iteratively converges on the desired result.
▸How-to Everyone
Not sure stack methods work correctly
Jest is a JavaScript testing framework that looks for files ending in *.test.js. By writing a failing test first (red), implementing just enough code to pass it (green), and then refactoring, you get immediate feedback and confidence that your code meets the specification.
▸How-to Everyone
Want to make sure a web form works for users
Cypress runs in a real browser and provides a jQuery‑like API to interact with the DOM. By scripting user actions (visit, type, click) and asserting UI state, you verify that the whole application works from the user's perspective.
▸How-to Everyone
Need a quick vanilla JavaScript starter
`npm init vite@latest my-app -- --template vanilla` quickly creates a minimal Vite‑powered project with an HTML entry point, dev server, and build pipeline, giving you a clean environment to start writing code and tests.
▸How-to Everyone
Want one config to declare models, prompts and test suites
The promptfoo.yml file is the entry point where you declare which LLM providers to use, reference prompt files, and specify test suites. Keeping this configuration simple lets you scale testing across many prompts without duplication.
▸How-to Everyone
Running a prompt with many variable inputs
Each YAML test defines a description, variables that replace placeholders in the prompt, and an `assertions` block that validates the model output. By swapping variable values you can generate many test cases from one template.
▸How-to Everyone
Not sure if LLM response is a proper SQL NLQ
Promptfoo offers several assertion types (string contains, regex, Levenshtein distance, custom JS/Python, and LLM rubric) that let you programmatically verify correctness, style, or sentiment of model responses.
▸How-to Everyone
Running `promptfoo eval` produces a table with latency (ms), token counts, and model identifiers for each test case. By examining these metrics you can pick the cheapest fastest model that still meets your assertions.
▸How-to Everyone
Need a new feature without writing tests yourself
By prompting an AI coding agent with a test‑first instruction, the agent first writes failing unit tests for a new feature, then implements just enough production code to make those tests pass. This mirrors classic TDD and ensures each change is verified before integration.
▸How-to Everyone
Same buyout‑price code in Java and Python backends
Opening separate AI coding windows for each language lets you run identical TDD prompts side‑by‑side, letting the AI handle language‑specific syntax while preserving a unified feature design across backends.
▸How-to Everyone
Backend adds a buyout price and the page breaks
When a new backend capability (e.g., buyout price) is added, writing front‑end unit or integration tests first catches mismatches like missing UI handling for auction closure, avoiding broken user flows.
▸How-to Everyone
Need unit tests for a function
Copilot can create unit test files for a selected function using the "/tests" slash command in Copilot Chat. It inspects the highlighted code, infers expected behavior, and writes both passing and failing test cases, saving you from writing boilerplate test scaffolding.
▸How-to Everyone
Want to add a tiny feature using test‑driven development
TDD means writing a failing test first (Red), implementing just enough code to make it pass (Green), then cleaning up the implementation (Refactor). Copilot speeds each step: it can generate the initial test, suggest minimal production code, and help refactor while keeping tests green.
▸How-to Everyone
Need a username checker but don’t want to code
By specifying validation rules in natural language, Copilot can generate both unit tests and the corresponding implementation for a brand‑new function, demonstrating how to prototype features without writing any code manually first.
▸How-to Everyone
Want clear guidance for AI‑generated code
A spec is a concise, behavior‑focused description of what the system should do, including endpoints, inputs, outputs and error handling. By defining this contract first, you give the LLM clear guidance, reducing guesswork and iteration.
▸How-to Everyone
Spec lists inputs, outputs and edge cases
Because the spec enumerates inputs, outputs and edge cases, you can programmatically turn each clause into a unit test. This ensures the AI‑generated code is validated against the original contract.
▸How-to Everyone
Spec is just a contract
The spec acts as a contract; converting it into a design document breaks the work into concrete to‑dos (e.g., create model, write controller). This mirrors traditional SDLC phases while still using an LLM for code creation.
▸How-to Everyone
Have a spec and need code, unit tests, and docs
Instead of prompting for ad‑hoc code snippets, you give the entire spec as context. The LLM can then output implementation, unit tests, and documentation that all align with the same contract.
The same set on /recipes, filtered by tool and role.
5Videos 4
Prompt Engineering Full Course 2026 | Generative AI | Prompt Engineering Tutorial| Simplilearn
This video delivers a comprehensive tutorial on prompt engineering for generative AI, teaching core techniques to structure, refine, and optimize prompts for reliable model outputs.
Test-Driven Development In Python // The Power of Red-Green-Refactor
Demonstrates test-driven development in Python using unittest features
Spec-Driven Development: AI Assisted Coding Explained
Explains how to turn specifications into design documents, use spec-driven prompting with LLMs to generate code, tests and documentation, and structure projects for AI-assisted development
Test Driven PROMPT Engineering: Using Promptfoo to COMPARE Prompts, LLMs, and Providers.
Shows how to set up Promptfoo configurations, create variable-driven prompt tests, and use its assertion features to validate LLM outputs
6FAQ 5
Why does one correct answer not prove my prompt works reliably?
A single successful output is just luck; it doesn’t show the prompt will handle variations in inputs. Reliability means the prompt consistently gives right answers across many situations, not just once.
How do I know what a correct AI response should look like before writing the prompt?
First write down the criteria for a “right” answer—specific wording, format, or content you expect. This definition guides you to craft a prompt that aims for those exact results instead of hoping for a lucky match.
What are input → expected output pairs and why should I create them?
They are concrete examples where you list an actual user input and the exact answer you want the AI to produce. Having these pairs lets you objectively test whether your prompt meets its goal.
Can you explain the red‑green‑refactor cycle for fixing prompts?
Start with a failing example (red), make the smallest change that makes it pass (green), then once all examples pass, tidy up the wording without breaking anything (refactor). This disciplined loop keeps edits minimal and the prompt stable.
How do I handle edge cases that previously broke my prompt?
Add those problematic scenarios as extra input → expected output pairs in your specification. By testing against these hard cases you force the prompt to handle them correctly.
7Glossary 12 terms
Show the 12 terms
Test-Driven Prompt Engineering
red‑green‑refactor cycle- A three‑step process where you first find a failing example (red), make the smallest change to fix it (green), then tidy up the prompt without breaking any tests (refactor).
Red- The label for an example that the current prompt gets wrong.
Green- The label for an example that the current prompt gets right after a change.
Refactor- A step where you simplify or reorganise the prompt while keeping all examples passing.
input → expected output- A pair showing what you will give the model and the exact answer you want it to produce.
edge case- An unusual or difficult example that previously caused the prompt to fail.
pass/fail grid- A table that marks each example as pass (green) or fail (red) after running the prompt.
example file- A document that lists all input‑output pairs and serves as the specification for testing the prompt.
defendability check- A quick test to see if you can explain why a prompt works after it fails once; if not, the prompt relies on guesswork.
regression run- Re‑executing all examples after a change to ensure previously passing cases are still correct.
delimiter- A character or string added to the prompt to clearly separate parts of the input, helping the model understand the structure.
model swap regression check- Running the same test suite on a different language‑model version to spot any new failures caused by the change.
8See also
💬 Discuss this chapter
Ask, share, or report — over on the Heidelberg AI community forum.