Transformers.js

The Transformers.js provider enables fully local inference using Transformers.js v4, running ONNX-optimized models directly in Node.js without external APIs or GPU setup. v4 features a new WebGPU backend, broader model support (8B+ parameter models), and improved performance.

Installation

Transformers.js is an optional dependency (~200MB for ONNX runtime):

npm install @huggingface/transformers

Quick Start

Embeddings

providers:
  - transformers:feature-extraction:Xenova/all-MiniLM-L6-v2

Popular models: Xenova/all-MiniLM-L6-v2 (384d), onnx-community/all-MiniLM-L6-v2-ONNX (384d), Xenova/bge-small-en-v1.5 (384d), nomic-ai/nomic-embed-text-v1.5 (768d)

Text Generation

providers:
  - transformers:text-generation:Xenova/gpt2

Popular models: Xenova/gpt2, onnx-community/Qwen3-0.6B-ONNX, onnx-community/Llama-3.2-1B-Instruct-ONNX

note

Text generation runs on CPU and is best for testing. For production, consider Ollama or cloud APIs.

Configuration

Common Options

These options apply to both embedding and text generation providers:

Option	Description	Default
device	'auto', 'cpu', 'gpu', 'wasm', 'webgpu', 'cuda', 'dml', 'coreml', 'webnn'	'auto'
dtype	Quantization: 'fp32', 'fp16', 'q8', 'q4', 'q4f16'	'auto'
cacheDir	Override model cache directory	System default
localFilesOnly	Skip downloads, use cached models only	false
revision	Model version/branch	'main'

Embedding Options

providers:
  - id: transformers:feature-extraction:Xenova/bge-small-en-v1.5
    config:
      prefix: 'Represent this sentence for searching relevant passages: ' # BGE retrieval queries
      pooling: cls # BGE v1.5 uses the CLS token embedding
      normalize: true # L2 normalize embeddings
      dtype: q8

Model prefixes: Follow the model card for your embedding model. BGE v1.5 uses the instruction above for retrieval queries; documents need no prefix. E5 v2 uses prefix: 'query: ' for queries and prefix: 'passage: ' for documents. MiniLM models need no prefix.

Nomic Embed v1.5 requires prefix: 'search_query: ' for retrieval queries and prefix: 'search_document: ' for indexed documents. Use the model card's task prefix for other workloads. Keep query and document preprocessing compatible with your vector index; rebuild affected stored embeddings when changing document preprocessing, model, or dimensions.

tip

transformers:embeddings:<model> is an alias for transformers:feature-extraction:<model>.

Text Generation Options

providers:
  - id: transformers:text-generation:onnx-community/Qwen3-0.6B-ONNX
    config:
      maxNewTokens: 256
      temperature: 0.7
      topK: 50
      topP: 0.9
      doSample: true
      repetitionPenalty: 1.1
      noRepeatNgramSize: 3
      numBeams: 1
      returnFullText: false
      dtype: q4

Using for Similarity Assertions

Use local embeddings as a grading provider for similar assertions:

defaultTest:
  options:
    provider:
      embedding:
        id: transformers:feature-extraction:Xenova/all-MiniLM-L6-v2

providers:
  - openai:gpt-4o-mini

tests:
  - vars:
      question: 'What is photosynthesis?'
    assert:
      - type: similar
        value: 'Photosynthesis converts light to chemical energy in plants'
        threshold: 0.8

Or override per-assertion:

assert:
  - type: similar
    value: 'Expected output'
    threshold: 0.75
    provider: transformers:feature-extraction:Xenova/all-MiniLM-L6-v2

Performance

• Caching: Pipelines are cached after first load. Initial model download may take time, but subsequent runs are fast.
• Quantization: Use dtype: q4 or dtype: q8 for faster inference and lower memory. Use dtype: q4f16 for WebGPU-optimized quantization.
• WebGPU: v4 includes a new WebGPU runtime written in C++ with significantly improved performance. Use device: webgpu on supported systems.
• Concurrency: For limited RAM, use promptfoo eval -j 1 to run serially.

Troubleshooting

Problem	Solution
Dependency not installed	Run npm install @huggingface/transformers
Model not found	Verify model exists at HuggingFace with ONNX weights. Try Xenova or onnx-community models.
Out of memory	Use dtype: q4, run with -j 1, or try smaller models
Slow first run	Models download on first use. Pre-download with await pipeline('feature-extraction', 'model-name')

Supported Models

Browse compatible models at huggingface.co/models?library=transformers.js.

Key organizations: onnx-community (optimized ONNX exports, recommended for v4), Xenova (legacy ONNX models, still compatible)