promptfoo/promptfoo
promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Usage guide
promptfoo is an open-source project around ci, ci-cd, cicd with 22,698 GitHub stars. This guide focuses on when to use it, how to install it, how to run the first example, and what to verify before adopting it.
Key features
- Implemented mainly in TypeScript, useful for judging integration effort in a similar stack.
- GitHub detected the MIT repository license, which generally permits commercial use. This signal only covers the repository license; review its obligations and any model weights, datasets, dependencies, or external services before commercial adoption.
- The project has a homepage, so cross-check docs, examples, and release information beyond GitHub.
Best for
- Evaluating promptfoo for TypeScript AI workflows.
- Comparing a GitHub project with 22,698 stars and current repository activity.
Pros
- promptfoo has visible GitHub traction with 22,698 stars. Topics: ci, ci-cd, cicd.
- The project provides an external homepage for deeper evaluation.
Cons
- Production fit still depends on documentation depth, issue activity, and release cadence.
- License review should confirm the MIT terms fit your use case.
Production readiness
promptfoo should be validated with its README, release history, open issues, and integration requirements before production use.
License risk
MIT is reported by GitHub; review the repository license before redistribution or commercial use.
promptfoo architecture preview
promptfoo's main path starts at the entry surface, runs through Retrieval pipeline, combines OpenAI / Claude / Gemini / DeepSeek, Files / repository context, Discord / Shell commands, and returns Grounded answers / search results.
Entry
CLI / terminal entry
promptfoo is primarily entered through a developer command or terminal workflow.
npm install -g promptfoo
Runtime
Retrieval pipeline
The pipeline retrieves relevant context before the model generates an answer.
RAG / retrieval
Model
OpenAI / Claude / Gemini / DeepSeek
Model calls are likely routed through OpenAI, Claude, Gemini, DeepSeek based on README and topic signals.
OpenAI, Claude, Gemini, DeepSeek
Context
Files / repository context
Context comes from Files / repository context, which constrains what the model or runtime can use.
Files / repository context
Tools
Discord / Shell commands
Tool adapters let the runtime act outside the model through Discord / Shell commands.
Discord, Shell commands
Output
Grounded answers / search results
The final result is an answer or ranked result grounded in retrieved context.
answer output
Featured video
Jason Koo
Start with PromptFoo in under 10 min.
12,225 views ยท 2025-04-06
Install tutorial
Before you install
- Python runtime and an isolated virtual environment
- Node.js and the package manager used by the project
- A clean working directory for the first test run
Check the runtime environment
promptfoo depends on a Python-style environment. Use venv, conda, or a container to keep dependencies isolated.
Get the project files
Start from the official repository or package so the first run matches the documented behavior.
$ git clone https://github.com/promptfoo/promptfoo.gitInstall or build dependencies
Run the next setup command detected from the project documentation.
$ npm install -g promptfooAdoption guidance and sources
Practical use cases
Knowledge-base assistant
Use it for document-grounded AI workflows where retrieval quality matters.
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerabil
This is one of the documented reasons to evaluate promptfoo before choosing a stack.
Focus area: ci
This is one of the documented reasons to evaluate promptfoo before choosing a stack.
RAG project comparison
Compare promptfoo with similar projects before committing to a stack.
Before adopting
- Complete one clean-environment verification using the official promptfoo setup path.
- Review repository license, model weights, external services, and dependency terms for your use case.
- Check recent commits, release cadence, issue response, and documentation depth.
- Evaluate output quality, latency, resource usage, and recovery behavior with a small dataset.
Configuration notes
- Review README configuration notes before using production data.
Sources checked
These links are used to verify repository, documentation, or tutorial details. Review the source pages before adopting the project.
Troubleshooting
- If installation fails, first confirm the command is being run from the README-specified directory.
- If dependencies conflict, retry in a fresh virtual environment, container, or working directory.
- If output looks wrong, return to the smallest documented promptfoo example before adding complex data.
- For keys, model files, or external services, verify environment variables, local paths, and permissions one by one.
- Before production use, review recent updates, open issues, license terms, and safety boundaries.
What is promptfoo?
promptfoo is an open-source rag project. Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
How do I install promptfoo?
Start with the official README. The first detected setup step is: git clone https://github.com/promptfoo/promptfoo.git.
Is promptfoo beginner-friendly?
If you already know the TypeScript ecosystem, start with the smallest example. Otherwise test it in an isolated environment first.
Can promptfoo be used commercially?
GitHub detected the MIT repository license, which generally permits commercial use. This signal only covers the repository license; review its obligations and any model weights, datasets, dependencies, or external services before commercial adoption.
Does promptfoo need a GPU?
GPU requirements depend on the workload, model, and dataset size. Start with the smallest README example before scaling up.
How should I decide whether to adopt promptfoo?
Evaluate setup cost, maintenance activity, issue health, license terms, and fit with your real workflow.