The GitHub Action for Promptfoo. Test your prompts, agents, and RAGs. AI Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.

Github Action for LLM Prompt Evaluation

This Github Action uses promptfoo to produce a before/after view of edit prompts.

When you change a prompt, an eval will automatically be posted on the pull request:

The provided link opens the promptfoo web viewer, which allows you to interactively explore the before vs. after:

Supported Events

This action supports multiple GitHub event types:

• Pull Request (pull_request, pull_request_target) - Compares changes between base and head branches
• Push (push) - Compares changes between commits (requires v1.1.0+)
• Manual Trigger (workflow_dispatch) - Allows manual evaluation with custom inputs (requires v1.1.0+)

Note: Version v1.0.0 only supports pull_request events. To use push or workflow_dispatch events, please use @v1 (which now points to v1.1.0+) or explicitly use @v1.1.0.

Configuration

The action can be configured using the following inputs: