airflow.providers.databricks.operators.delta_table

Produce asset events for Delta table versions committed outside of Airflow.

Attributes

CHECKPOINT_KEY_EXTRA

Asset event extra key that marks an event as a checkpoint written by this producer.

DEFAULT_IGNORED_OPERATIONS

ChangeKind

Classes

DeltaCheckpoint

Last Delta version this producer published for a table, read back from an asset event.

DatabricksDeltaTableVersionProducerOperator

Publish an asset event when a Unity Catalog Delta table gets a newer committed version.

Module Contents

airflow.providers.databricks.operators.delta_table.CHECKPOINT_KEY_EXTRA = 'databricks_delta_checkpoint_key'[source]

Asset event extra key that marks an event as a checkpoint written by this producer.

airflow.providers.databricks.operators.delta_table.DEFAULT_IGNORED_OPERATIONS = ('OPTIMIZE', 'VACUUM START', 'VACUUM END')[source]
airflow.providers.databricks.operators.delta_table.ChangeKind[source]
class airflow.providers.databricks.operators.delta_table.DeltaCheckpoint[source]

Last Delta version this producer published for a table, read back from an asset event.

version: int[source]
table_id: str | None[source]
class airflow.providers.databricks.operators.delta_table.DatabricksDeltaTableVersionProducerOperator(*, unity_table, checkpoint_key=None, baseline_version=None, ignored_operations=None, max_history_rows=1000, on_recreation='publish', databricks_conn_id=DatabricksSqlHook.default_conn_name, http_path=None, sql_warehouse_name=None, session_configuration=None, http_headers=None, client_parameters=None, query_tags=None, include_airflow_query_tags=True, **kwargs)[source]

Bases: airflow.providers.common.compat.sdk.BaseOperator

Publish an asset event when a Unity Catalog Delta table gets a newer committed version.

Use this operator in a recurring producer Dag to turn Delta commits made by other jobs or systems into Airflow asset events. Downstream Dags schedule on unity_table.to_asset().

Each run reads the newest Delta version, compares it with the checkpoint, and either publishes one asset event for the newest version or skips. Several commits between two runs are coalesced into one event for the newest version. Intermediate commits are not delivered one by one.

The checkpoint is the newest asset event that this producer published for the asset, found by checkpoint_key in the event extra. Airflow records the event when the task succeeds, so publishing the event and advancing the checkpoint are the same write. Delivery is at least once: concurrent runs or a cleared run can publish the same version again. Consumers that must not process a version twice deduplicate on extra["dedup_id"].

An event for version N does not pin downstream reads to N. A consumer that needs a reproducible read queries VERSION AS OF extra["version"].

See also

For more information on how to use this operator, take a look at the guide: DatabricksDeltaTableVersionProducerOperator

Parameters:
  • unity_table (airflow.providers.databricks.assets.databricks.UnityTableIdentity) – Static identity of the observed table. It is added to inlets (to read the checkpoint) and, when outlets is omitted, outlets is [unity_table.to_asset()]. Explicit outlets must include unity_table.to_asset(), because the checkpoint is the event published on that asset.

  • checkpoint_key (str | None) – Scope of the checkpoint. Producers with different keys keep separate checkpoints for the same asset. Defaults to "<dag_id>.<task_id>". Changing it starts a new enrollment.

  • baseline_version (int | None) – Version treated as already delivered when no checkpoint exists. When None (the default), the first run publishes an enrollment event for the current version.

  • ignored_operations (collections.abc.Collection[str] | None) – Delta history operations that do not count as a change. Defaults to OPTIMIZE, VACUUM START and VACUUM END. Pass an empty collection to publish for every commit.

  • max_history_rows (int) – Largest DESCRIBE HISTORY window read to prove that all commits after the checkpoint are ignored operations. When the window or the retained history does not reach the checkpoint, the operator publishes rather than risk missing a data change.

  • on_recreation (Literal['publish', 'fail']) – What to do when the table id differs from the checkpoint, or the version moved backwards, which means the table was dropped and recreated or replaced by a rename. "publish" publishes a recreated event and checkpoints the new table. "fail" raises and leaves the checkpoint untouched.

  • databricks_conn_id (str) – Reference to Databricks connection id.

  • http_path (str | None) – HTTP path of a Databricks SQL warehouse or all-purpose cluster.

  • sql_warehouse_name (str | None) – Name of a Databricks SQL warehouse, used when http_path is not set.

  • session_configuration (dict[str, str] | None) – Spark session parameters.

  • http_headers (list[tuple[str, str]] | None) – (k, v) pairs set as HTTP headers on every request.

  • client_parameters (dict[str, Any] | None) – Additional Databricks SQL connector parameters.

  • query_tags (dict[str, str | None] | None) – Query tags attached to the Databricks SQL queries. (templated)

  • include_airflow_query_tags (bool) – If True, add Airflow Dag/task/run metadata as query tags.

template_fields: collections.abc.Sequence[str] = ('databricks_conn_id', 'http_headers', 'query_tags')[source]
unity_table[source]
baseline_version = None[source]
ignored_operations = ('OPTIMIZE', 'VACUUM START', 'VACUUM END')[source]
max_history_rows = 1000[source]
on_recreation = 'publish'[source]
databricks_conn_id = 'databricks_default'[source]
http_path = None[source]
sql_warehouse_name = None[source]
session_configuration = None[source]
http_headers = None[source]
client_parameters[source]
query_tags[source]
include_airflow_query_tags = True[source]
property checkpoint_key: str[source]
property asset: airflow.providers.common.compat.assets.Asset[source]
property table_name: str[source]
property hook: airflow.providers.databricks.hooks.databricks_sql.DatabricksSqlHook[source]
execute(context)[source]

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

read_checkpoint(context)[source]

Return the newest checkpoint for checkpoint_key from the asset’s events, if any.

Was this entry helpful?