Airflow Summit 2026 is coming August 31 - September 2 in Austin, TX. Register now to secure your spot!

airflow.providers.anthropic.operators.batch

Classes

AnthropicBatchOperator

Submit an Anthropic Message Batch and wait for it to complete.

Module Contents

class airflow.providers.anthropic.operators.batch.AnthropicBatchOperator(requests, conn_id=AnthropicHook.default_conn_name, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), poll_interval=60, timeout=24 * 60 * 60, wait_for_completion=True, fail_on_partial_error=False, **kwargs)[source]

Bases: airflow.providers.common.compat.sdk.BaseOperator

Submit an Anthropic Message Batch and wait for it to complete.

Message Batches process many messages.create requests asynchronously at 50% of standard cost; most complete within an hour (24h SLA). This operator submits the batch and, in deferrable mode, releases the worker slot while a trigger polls for completion.

The operator returns the batch ID only — never the results. Pull results with stream_batch_results() and persist them to object storage; results can be very large and must not be pushed to XCom. Results are retained for 29 days after the batch is created.

Note

A retry re-submits a brand-new batch. Prefer retries=0 on this task (the submitted batch_id is pushed to XCom under key batch_id immediately, so a crashed run never loses track of an in-flight batch).

See also

For more information, take a look at the guide: AnthropicBatchOperator

Parameters:

  • requests (list[dict[str, Any]]) – A list of {"custom_id": str, "params": {...}} dicts, where params is a messages.create payload (model, max_tokens, messages, …).

  • conn_id (str) – The Anthropic connection ID to use.

  • deferrable (bool) – Run the operator in deferrable mode.

  • poll_interval (float) – Seconds between status checks, in both the synchronous and deferrable paths.

  • timeout (float) – Seconds to wait for the batch to reach a terminal status. Defaults to 24 hours (the Message Batches SLA). In deferrable mode this also bounds the deferral; set execution_timeout only if you want a shorter hard cap (note a shorter execution_timeout preempts the graceful cancel-on-timeout path).

  • wait_for_completion (bool) – Whether to wait for the batch to complete. If False, the operator returns the batch ID immediately after submission.

  • fail_on_partial_error (bool) – If True, fail the task when any request errored or expired. Defaults to False (succeed and log a warning so the successful results are not discarded).

template_fields: collections.abc.Sequence[str] = ('requests',)[source]
requests[source]
conn_id = 'anthropic_default'[source]
deferrable[source]
poll_interval = 60[source]
timeout = 86400[source]
wait_for_completion = True[source]
fail_on_partial_error = False[source]
batch_id: str | None = None[source]
property hook: airflow.providers.anthropic.hooks.anthropic.AnthropicHook[source]

Return an instance of the AnthropicHook.

execute(context)[source]

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]

Resume after the trigger fires.

The deferred task is a fresh instance, so the batch ID is read from the event, not self.batch_id.

on_kill()[source]

Cancel the batch if the (non-deferred) task is killed.

This only fires while the worker process is alive, i.e. the synchronous path (deferrable=False). On Airflow 3.3+ a killed deferred task is cancelled by the trigger’s on_kill. On older Airflow the batch of a killed deferred task is not cancelled automatically; cancel it manually via the hook.

Was this entry helpful?