Skip to main content

Agent Skills for Evals and Red Teaming

AI coding agents can write promptfoo configs, but they often get the details wrong: shell-style env vars that do not work, hallucination rubrics that cannot see the source material, tests dumped inline instead of in files, and red-team configs that collapse real app inputs into one generic prompt field.

Promptfoo ships one agent-skill bundle with four focused skills — promptfoo-evals for eval authoring, promptfoo-provider-setup for connecting targets, and promptfoo-redteam-setup plus promptfoo-redteam-run for red-team setup and scan triage. The same bundle is published to both the Claude Code and OpenAI Codex marketplaces.

It follows the open Agent Skills standard, so the skills should also work with other compatible tools.

Why use a skill?​

Without the skill, agents frequently:

  • Use $ENV_VAR syntax in YAML configs, which does not work because promptfoo uses Nunjucks '{{env.VAR}}'
  • Write llm-rubric assertions that reference "the article" but don't inline the source, so the grader can't actually compare
  • Dump all tests inline in the config instead of using file://tests/*.yaml
  • Reach for llm-rubric when contains or is-json would be faster, free, and deterministic

The skill gives the agent these rules up front.

The red-team skills cover a different set of common mistakes: flattening multi-input targets into one prompt field, choosing broad scans before mapping the app boundary, and regenerating probes when a stable rerun would be easier to compare.

Install​

Via Claude Code marketplace​

/plugin marketplace add promptfoo/promptfoo
/plugin install promptfoo@promptfoo

This installs all four skills. Ask the agent to create an eval, connect a target, or run a red team and it routes to the right skill, or invoke one directly with a namespaced slash command such as /promptfoo:promptfoo-evals.

note

This plugin was previously published as promptfoo-evals (eval skill only). If you installed it under that name, reinstall with /plugin install promptfoo@promptfoo to get the full four-skill bundle and future updates.

Via Codex plugin bundle​

For Codex, the same plugins/promptfoo bundle is exposed by .agents/plugins/marketplace.json. Add it to a Codex workspace to install the same four skills.

The four skills​

Both marketplaces install the same bundle at plugins/promptfoo, exposed by .claude-plugin/marketplace.json for Claude Code and .agents/plugins/marketplace.json for Codex:

SkillUse it for
promptfoo-evalsNon-redteam eval suites, assertions, test cases, and result inspection
promptfoo-provider-setupHTTP targets plus JavaScript or Python file:// providers and wrappers
promptfoo-redteam-setupFocused redteam configs from live endpoints, OpenAPI specs, or static code
promptfoo-redteam-runRunning generated scans, triaging failures, and filtered reruns

There is intentionally no meta selector skill. The agent routes from each skill's description and default prompt.

Python providers are first-class in the bundle. The provider and redteam skills cover Promptfoo's file://provider.py and file://provider.py:function_name syntax for eval providers, redteam targets, local graders, and local redteam generators, including workers, timeout, and PROMPTFOO_PYTHON configuration.

To reuse the bundle in another workspace, copy plugins/promptfoo together with its marketplace entry — .claude-plugin/marketplace.json for Claude Code or .agents/plugins/marketplace.json for Codex.

For red teaming, promptfoo-provider-setup connects the system under test, promptfoo-redteam-setup turns live endpoints, OpenAPI specs, or static code into a scan plan, and promptfoo-redteam-run executes and triages the generated probes.

Manual install​

For an eval-only setup, copy the self-contained promptfoo-evals skill into your project:

Claude Code (project-level, recommended for teams):

cp -r promptfoo-evals your-project/.claude/skills/

Claude Code (personal, available in all projects):

cp -r promptfoo-evals ~/.claude/skills/

OpenAI Codex / other Agent Skills tools:

cp -r promptfoo-evals your-project/.agents/skills/

To add provider setup or red teaming as well, install the full bundle from the marketplace (above) so the skills can hand off to each other, or copy the whole plugins/promptfoo/skills directory so the referenced sibling skills resolve.

note

Commit skills to .claude/skills/ or .agents/skills/ so every developer's agent picks them up automatically, with no per-person install needed.

Each skill consists of a SKILL.md with workflow instructions plus a references/ directory of assertion types, provider patterns, and config examples (provider and redteam setup also include a scripts/ directory).

Usage​

Once installed, the agent activates automatically when you ask it to create or update eval coverage. In Claude Code, you can also invoke a skill directly with a slash command (namespaced when installed from the marketplace):

/promptfoo:promptfoo-evals Create an eval suite for my summarization prompt

In Codex and other Agent Skills tools, ask the agent to create an eval. The skill activates from the task context.

For red-team work, ask for the task directly:

Create a focused red team config for this invoice assistant. Preserve user_id, invoice_id, and message inputs; test policy, RBAC, and BOLA.
Run the generated redteam scan, summarize attack success rate, and give me the narrowest rerun command for failures.

The agent:

  1. Search for existing promptfoo configs in the repo
  2. Scaffold a new suite if needed (promptfooconfig.yaml, prompts/, tests/)
  3. Write test cases with deterministic assertions first, model-graded when needed
  4. Validate the config with promptfoo validate
  5. Provide run commands
note

New to promptfoo? See Getting Started for an overview of configs, providers, and assertions.

What the skill teaches​

  • Deterministic assertions first. Use contains, is-json, javascript before reaching for llm-rubric. Deterministic checks are fast, free, and reproducible.
  • File-based test organization. Tests go in tests/*.yaml files loaded via file://tests/*.yaml glob, keeping configs clean as test count grows.
  • Dataset-driven scaling. For larger suites, use tests: file://tests.csv or script-generated tests like file://generate_tests.py:create_tests.
  • Faithfulness checks done right. When using llm-rubric to check for hallucination, the source material must be inlined in the rubric via {{variable}} so the grader can actually compare.
  • Pinned grader provider. Model-graded assertions should explicitly set a grading provider (defaultTest.options.provider or assertion.provider) for stable scoring.
  • Environment variables. Use Nunjucks syntax '{{env.API_KEY}}' in YAML configs, not shell syntax.
  • CI-friendly runs. Use promptfoo eval -o output.json --no-cache and inspect success, score, and error.
  • Config field ordering. description, env, prompts, providers, defaultTest, scenarios, tests.

The provider and red-team skills also teach the agent to:

  • Keep real inputs such as user IDs, object IDs, documents, and tools visible so authorization and agent-boundary issues stay testable.
  • Choose plugins such as policy, rbac, bola, hijacking, prompt-extraction, and system-prompt-override from live or static evidence instead of defaulting to one broad scan.
  • Inspect generated probes before running them, reuse generated tests with redteam eval when possible, and separate grader failures from real target failures.
  • Prefer no-share runs for internal systems and keep provider secrets in environment variables rather than committed configs.

Example output​

Ask the agent to "create an eval for a customer support chatbot that returns JSON" and it produces:

promptfooconfig.yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: 'Customer support chatbot'

prompts:
  - file://prompts/chat.json

providers:
  - id: openai:chat:gpt-4.1-mini
    config:
      temperature: 0
      response_format:
        type: json_object

defaultTest:
  assert:
    - type: is-json
    - type: cost
      threshold: 0.01

tests:
  - file://tests/*.yaml
tests/happy-path.yaml
- description: 'Returns order status for valid customer'
  vars:
    order_id: 'ORD-1001'
    customer_name: 'Alice Smith'
  assert:
    - type: is-json
      value:
        type: object
        required: [status, message]
    - type: javascript
      value: "JSON.parse(output).status === 'shipped'"

A red-team setup should keep the security boundary visible instead of collapsing it into one free-form prompt:

promptfooconfig.yaml
description: 'Invoice assistant red team'

targets:
  - id: https
    label: invoice-assistant
    inputs:
      user_id: Signed-in user identifier.
      invoice_id: Invoice being requested.
      message: User message.
    config:
      url: '{{env.INVOICE_AGENT_URL}}'
      method: POST
      stateful: false
      body:
        user_id: '{{user_id}}'
        invoice_id: '{{invoice_id}}'
        message: '{{message}}'
      transformResponse: json.output

redteam:
  purpose: >-
    Invoice assistant for signed-in users. It may answer questions about the
    caller's invoices only and must not reveal or modify other users' invoices.
  plugins:
    - id: policy
      config:
        policy: The assistant must not disclose or modify another user's invoices.
    - rbac
    - bola
  strategies:
    - basic

Customizing the skill​

The skill is just markdown files. Edit them to match your team's conventions:

  • Add custom providers to the reference files if your team uses specific models or endpoints.
  • Add assertion patterns for your domain (e.g., medical accuracy rubrics, financial compliance checks).
  • Change the default layout if your repo uses a different directory structure for evals.
Last updated on by renovate[bot]