Quick start¶

Go from zero to a submitted Claude batch in three steps: install the provider, configure a connection, and write a Dag around AnthropicBatchOperator. This provider is built around the Message Batches API and Managed Agent sessions; for interactive, single-call LLM tasks, use apache-airflow-providers-common-ai instead.

1. Install¶

pip install apache-airflow-providers-anthropic

2. Configure the connection¶

Batches run through an Anthropic connection (conn_type anthropic, default connection id anthropic_default). For the first-party API, put your Anthropic API key in the password field. See Anthropic Connection for the full reference, including the platform extra for running on Amazon Bedrock, Google Vertex AI, Claude Platform on AWS or Microsoft Foundry, and for keyless auth via Workload Identity Federation.

The quickest way to set one up is an environment variable:

export AIRFLOW_CONN_ANTHROPIC_DEFAULT='{"conn_type": "anthropic", "password": "sk-ant-..."}'

Or add it through the Airflow UI (Admin > Connections) or the CLI (airflow connections add).

3. Write your first Dag¶

AnthropicBatchOperator submits a Message Batch — a list of messages.create requests — and waits for it to reach a terminal status. A task retry resubmits a new batch, so set retries=0:

airflow/providers/anthropic/example_dags/example_batch.py[source]

from __future__ import annotations

from airflow.providers.anthropic.operators.batch import AnthropicBatchOperator
from airflow.sdk import dag


@dag(tags=["example"])
def quickstart_batch():
    AnthropicBatchOperator(
        task_id="submit_batch",
        requests=[
            {
                "custom_id": "summary-1",
                "params": {
                    "model": "claude-opus-4-8",
                    "max_tokens": 1024,
                    "messages": [
                        {"role": "user", "content": "Summarize the plot of Hamlet in two sentences."}
                    ],
                },
            },
            {
                "custom_id": "summary-2",
                "params": {
                    "model": "claude-opus-4-8",
                    "max_tokens": 1024,
                    "messages": [
                        {"role": "user", "content": "Summarize the plot of Macbeth in two sentences."}
                    ],
                },
            },
        ],
        wait_for_completion=True,
        retries=0,
    )


quickstart_batch()

Run it like any other Dag (airflow dags test quickstart_batch). The task pushes the batch ID to XCom under key batch_id as soon as it submits, and returns it once the batch reaches ended. Pull the per-request results with stream_batch_results() and write them to object storage — results can be very large and must not be pushed to XCom.

Where to go next¶

Anthropic Operators — full parameter reference for AnthropicBatchOperator, AnthropicBatchSensor (poll an already-submitted batch without resubmitting on retry) and AnthropicAgentSessionOperator (Anthropic-hosted Managed Agent sessions).
Anthropic Connection — the platform extra (bedrock, vertex, aws, foundry) and Workload Identity Federation for keyless auth.
For interactive, single-call or agentic LLM workloads, prefer the vendor-agnostic apache-airflow-providers-common-ai provider with model="anthropic:claude-opus-4-8"; this provider focuses on the batch/async surface and direct SDK access that the agent abstraction does not model.