Databricks Foundation Model APIs

The databricks: provider sends chat requests through Databricks' OpenAI-compatible Foundation Model APIs. It supports pay-per-token, provisioned, and external chat endpoints. Use the exact serving endpoint name as the provider suffix; custom endpoint names are workspace-specific identities.

Overview

Databricks Foundation Model APIs provide three main deployment options:

1. Pay-per-token endpoints: Pre-configured endpoints for popular models with usage-based pricing
2. Provisioned throughput: Dedicated endpoints with guaranteed performance for production workloads
3. External models: Unified access to models from providers like OpenAI, Anthropic, and Google through Databricks

Prerequisites

1. A Databricks workspace with Foundation Model APIs enabled
2. A Databricks access token for authentication
3. Your workspace URL (e.g., https://your-workspace.cloud.databricks.com)

Set up your environment:

export DATABRICKS_WORKSPACE_URL=https://your-workspace.cloud.databricks.com
export DATABRICKS_TOKEN=your-token-here

Basic Usage

Pay-per-token Endpoints

Access pre-configured Foundation Model endpoints with simple configuration:

promptfooconfig.yaml

providers:
  - id: databricks:databricks-meta-llama-3-3-70b-instruct
    config:
      isPayPerToken: true
      workspaceUrl: https://your-workspace.cloud.databricks.com

Example pay-per-token chat endpoints include:

• databricks-meta-llama-3-3-70b-instruct - Meta Llama 3.3 70B Instruct
• databricks-claude-sonnet-4-6 - Anthropic Claude Sonnet 4.6

Check the current model catalog and retirement policy for availability. The pay-per-token offerings for Claude 3.7 Sonnet and DBRX have retired; this does not rename custom endpoints in your workspace.

Databricks also offers embedding models such as databricks-gte-large-en, but the databricks: provider is chat-only. Embedding requests require a separate integration with the Databricks embeddings API.

Provisioned Throughput Endpoints

For production workloads requiring guaranteed performance:

providers:
  - id: databricks:my-custom-endpoint
    config:
      workspaceUrl: https://your-workspace.cloud.databricks.com
      temperature: 0.7
      max_tokens: 500

External Models

Access external models through Databricks' unified API:

providers:
  - id: databricks:my-openai-endpoint
    config:
      workspaceUrl: https://your-workspace.cloud.databricks.com
      # External model endpoints proxy to providers like OpenAI, Anthropic, etc.

Configuration Options

The Databricks provider extends the OpenAI configuration options with these Databricks-specific features:

Parameter	Description	Default
workspaceUrl	Databricks workspace URL. Can also be set via DATABRICKS_WORKSPACE_URL environment variable	-
isPayPerToken	Legacy classification; both values use the OpenAI-compatible chat endpoint	false
usageContext	Optional metadata for usage tracking and cost attribution	-

Advanced Configuration

promptfooconfig.yaml

providers:
  - id: databricks:databricks-claude-sonnet-4-6
    config:
      isPayPerToken: true
      workspaceUrl: https://your-workspace.cloud.databricks.com

      # Standard OpenAI parameters
      temperature: 0.7
      max_tokens: 2000

      # Usage tracking for cost attribution
      usageContext:
        project: 'customer-support'
        team: 'engineering'
        environment: 'production'

Configure safety filters and PII handling on the Databricks serving endpoint. The legacy aiGatewayConfig option does not enable these controls.

Both pay-per-token and custom chat endpoints use /serving-endpoints/chat/completions, with the serving endpoint name in the model field.

Environment Variables

Variable	Description
DATABRICKS_WORKSPACE_URL	Your Databricks workspace URL
DATABRICKS_TOKEN	Authentication token for Databricks API access

Features

Vision Models

Vision models on Databricks require structured JSON prompts similar to OpenAI's format. Here's how to use them:

promptfooconfig.yaml

prompts:
  - file://vision-prompt.json

providers:
  - id: databricks:databricks-claude-sonnet-4-6
    config:
      isPayPerToken: true

tests:
  - vars:
      question: "What's in this image?"
      image_url: 'https://example.com/image.jpg'

Create a vision-prompt.json file with the proper format:

vision-prompt.json

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "{{question}}"
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "{{image_url}}"
        }
      }
    ]
  }
]

Structured Outputs

Get responses in a specific JSON schema:

providers:
  - id: databricks:databricks-meta-llama-3-3-70b-instruct
    config:
      isPayPerToken: true
      response_format:
        type: 'json_schema'
        json_schema:
          name: 'product_info'
          schema:
            type: 'object'
            properties:
              name:
                type: 'string'
              price:
                type: 'number'
            required: ['name', 'price']

Monitoring and Usage Tracking

Track usage and costs with detailed context:

providers:
  - id: databricks:databricks-meta-llama-3-3-70b-instruct
    config:
      isPayPerToken: true
      usageContext:
        application: 'chatbot'
        customer_id: '12345'
        request_type: 'support_query'
        priority: 'high'

Usage data is available through Databricks system tables:

• system.serving.endpoint_usage - Token usage and request metrics
• system.serving.served_entities - Endpoint metadata

Best Practices

1. Choose the right deployment mode:

   • Use pay-per-token for experimentation and low-volume use cases
   • Use provisioned throughput for production workloads requiring SLAs
   • Use external models when you need specific providers' capabilities

2. Enable AI Gateway features for production endpoints:

   • Safety guardrails prevent harmful content
   • PII detection protects sensitive data
   • Rate limiting controls costs and prevents abuse

3. Implement proper error handling:

   • Pay-per-token endpoints may have rate limits
   • Provisioned endpoints may have token-per-second limits
   • External model endpoints inherit provider-specific limitations

Example: Multi-Model Comparison

promptfooconfig.yaml

prompts:
  - 'Explain quantum computing to a 10-year-old'

providers:
  # Databricks native model
  - id: databricks:databricks-meta-llama-3-3-70b-instruct
    config:
      isPayPerToken: true
      temperature: 0.7

  # External model via Databricks
  - id: databricks:my-gpt4-endpoint
    config:
      temperature: 0.7

  # Custom deployed model
  - id: databricks:my-finetuned-llama
    config:
      temperature: 0.7

tests:
  - assert:
      - type: llm-rubric
        value: 'Response should be simple, clear, and use age-appropriate analogies'

Troubleshooting

Common issues and solutions:

1. Authentication errors: Verify your DATABRICKS_TOKEN has the necessary permissions
2. Endpoint not found:
   • For pay-per-token: Ensure you're using the exact endpoint name (e.g., databricks-meta-llama-3-3-70b-instruct)
   • For custom endpoints: Verify the endpoint exists and is running
3. Rate limiting: Pay-per-token endpoints have usage limits; consider provisioned throughput for high-volume use
4. Token count errors: Some models have specific token limits; adjust max_tokens accordingly

Additional Resources

• Databricks Foundation Model APIs documentation
• Supported models and regions
• AI Gateway configuration
• Unity Catalog model management