Promptfoo is a free, open-source tool for testing LLM prompts and applications against a defined set of cases and assertions, with red-team vulnerability scanning built in. Pick it when you want prompt changes to run through the same test set every time, not just a quick manual check. Community edition is free forever with all evaluation features; Enterprise adds team collaboration, SSO and on-premise deployment.
Other good fits
Red-teaming an LLM application for vulnerabilities as part of your development process, not as a one-off audit.
+ Pros
Read pros
Promptfoo runs your prompts or LLM application against a defined set of test cases and assertions, so you get a repeatable pass/fail result instead of eyeballing outputs each time something changes. The Community edition is free and open source, with all LLM evaluation features, support for all model providers and integrations, local or self-hosted deployment, and red-team vulnerability scanning (capped at 10,000 probes/month).
Pick it when…
You want to catch a prompt regression automatically instead of manually re-checking outputs after every change.
You want red-team vulnerability scanning (testing for prompt injection, jailbreaks and similar issues) as a routine part of development.
You want to self-host and keep test data local, without a paid tier.
− Cons
Read cons
Results are only as good as the test cases and assertions you actually write; Promptfoo doesn't know what "correct" means for your application, so a shallow test set will pass things it shouldn't and miss real regressions. Running tests against a live model API means every test run has a real token cost, which adds up if your CI runs the full suite on every commit without any caching or sampling strategy.
Pick something else when…
You want observability on production traffic, not pre-deployment testing.Langfuse is built for tracing and scoring real usage, which is a different (complementary) problem from Promptfoo's test-before-you-ship focus.
You need more than 10,000 red-team probes a month without paying. Check Enterprise pricing directly, since that's the only route past the Community cap.
Your testing needs are simple enough for manual spot-checks. For a project not yet iterating on prompts regularly, formal test infrastructure may be premature.
Avoid if
You need more than 10,000 red-team probes a month; the free Community tier caps at that, and higher volume needs Enterprise.
Workflow
Define a set of test cases (representative inputs and what a correct or acceptable output looks like) → write assertions checking for that → run promptfoo against your current prompt or application → review pass/fail results and specific failures → fix the prompt and rerun → add the failure case to your test set so it's caught automatically next time.
All evaluation features, all providers, local/self-hosted, red teaming up to 10k probes/month
Enterprise
Custom
Higher red-team limits, team collaboration, SSO, API access, managed cloud, SLA support
Enterprise On-Premise
Custom
All Enterprise features, your own infrastructure, complete data isolation, dedicated deployment engineer
The Community tier costs nothing beyond your own compute and whatever model-provider API calls your tests make (each test run calls the actual model, which has its own token cost). Running effort is mostly in writing good test cases and assertions up front; a thin test set gives you false confidence just as much as no tests would.
Restrictions
The free Community tier caps red-team probes at 10,000/month; higher volume needs Enterprise. Self-hosted deployment, provider API keys, and CI integration are your own responsibility, not something Promptfoo manages for you.
First project
Documentation-based suggestion — not run here.
Build a small regression test for one prompt.
Take a prompt you're actively iterating on, and write five test cases covering both typical inputs and at least one edge case.
Write an assertion for each (a keyword that must appear, a format check, or an LLM-graded rubric).
Run promptfoo against your current prompt version and confirm all five pass.
Deliberately break something in the prompt and rerun, confirming the test catches the regression.
Done when your test set catches a deliberately introduced regression, proving the tests actually check something meaningful rather than always passing.
Links & evidence
Documentation-based · not hands-on · Sep 23, 2026
Rests on Promptfoo's pricing page and documentation, checked 23 September 2026. No test suite was built or run for this review.
Promptfoo pricing · Sep 23, 2026 — Community (free), Enterprise, Enterprise On-Premise tiers; 10k probes/month cap on free red teaming.
Promptfoo documentation · Sep 23, 2026 — Test case and assertion workflow, evaluation concepts.
Promptfoo · Sep 23, 2026 — Product homepage.
Further review notes & alternatives
Alternatives and trade-offs
Langfuse wins when you need to observe and score real production traffic, complementing Promptfoo's pre-deployment testing rather than replacing it.
Manual spot-checking wins for a project not yet iterating on prompts often enough to justify formal test infrastructure.
Enterprise Promptfoo wins when you need red-team volume beyond the free tier's 10,000 probes/month, or team collaboration and SSO.
Define repeatable evaluations for prompts, models, providers, and datasets.
Add assertions that turn expected behavior into automated checks.
Run regression tests to compare changes and expose failures over time.
Owner-approved use of this public official repository visual, 13 September 2026. Source attribution retained. Converted on 15 October 2026 to a 1600px-wide WebP (quality 80) for faster page loads; the original image remains in the media library.