Skip to main content

HuggingFace Datasets

Promptfoo can import test cases directly from HuggingFace datasets using the huggingface://datasets/ prefix.

Basic usage​

To load an entire dataset:

tests: huggingface://datasets/fka/awesome-chatgpt-prompts

Run the evaluation:

npx promptfoo eval

Each dataset row becomes a test case with all dataset fields available as variables.

Dataset splits​

Load specific portions of datasets using query parameters:

# Load from training split
tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=train

# Load from validation split with custom configuration
tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=validation&config=custom

Use dataset fields in prompts​

Dataset fields automatically become prompt variables. Here's how:

promptfooconfig.yaml
prompts:
  - "Question: {{question}}\nAnswer:"

tests: huggingface://datasets/rajpurkar/squad

Query parameters​

ParameterDescriptionDefault
splitDataset split to load (train/test/validation)test
configDataset configuration (also called a subset)default
limitMaximum number of test cases to loadunlimited

The loader uses the Hugging Face dataset viewer /rows API. Promptfoo manages offset and length for pagination; use limit to cap the total number of test cases.

To limit the number of test cases:

tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=train&limit=50

To load a specific subset (common with MMLU datasets), set config:

tests: huggingface://datasets/cais/mmlu?split=test&config=college_physics&limit=10

Authentication​

For private datasets or increased rate limits, authenticate using your HuggingFace token. Set one of these environment variables:

# Any of these environment variables will work:
export HF_TOKEN=your_token_here
export HF_API_TOKEN=your_token_here
export HUGGING_FACE_HUB_TOKEN=your_token_here
info

Authentication is required for private datasets and gated models. For public datasets, authentication is optional but provides higher rate limits.

Implementation details​

  • Each dataset row becomes a test case
  • All dataset fields are available as prompt variables
  • Large datasets are automatically paginated (100 rows per request)
  • Variable expansion is disabled to preserve original data

Example configurations​

Basic chatbot evaluation​

promptfooconfig.yaml
description: Testing with HuggingFace dataset

prompts:
  - 'Act as {{act}}. {{prompt}}'

providers:
  - openai:gpt-5-mini

tests: huggingface://datasets/fka/awesome-chatgpt-prompts?split=train

Question answering with limits​

promptfooconfig.yaml
description: SQUAD evaluation with authentication

prompts:
  - 'Question: {{question}}\nContext: {{context}}\nAnswer:'

providers:
  - openai:gpt-5-mini

tests: huggingface://datasets/rajpurkar/squad?split=validation&limit=100

env:
  HF_TOKEN: your_token_here

Example projects​

ExampleUse CaseKey Features
Basic SetupSimple evaluationDefault parameters
MMLU-Pro ComparisonQuery parametersSplit, config, limit
Red Team SafetySafety testingBeaverTails dataset

Troubleshooting​

Authentication errors​

Ensure your HuggingFace token is set correctly: export HF_TOKEN=your_token

Dataset not found​

Verify the dataset path format: owner/repo (e.g., rajpurkar/squad)

Empty results​

Check that the specified split exists for the dataset. Try split=train if split=test returns no results.

Performance issues​

Add the limit parameter to reduce the number of rows loaded: &limit=100

See Also​

Last updated on by renovate[bot]