airflow.providers.databricks.operators.delta_table¶
Produce asset events for Delta table versions committed outside of Airflow.
Attributes¶
Asset event |
|
Classes¶
Last Delta version this producer published for a table, read back from an asset event. |
|
Publish an asset event when a Unity Catalog Delta table gets a newer committed version. |
Module Contents¶
- airflow.providers.databricks.operators.delta_table.CHECKPOINT_KEY_EXTRA = 'databricks_delta_checkpoint_key'[source]¶
Asset event
extrakey that marks an event as a checkpoint written by this producer.
- airflow.providers.databricks.operators.delta_table.DEFAULT_IGNORED_OPERATIONS = ('OPTIMIZE', 'VACUUM START', 'VACUUM END')[source]¶
- class airflow.providers.databricks.operators.delta_table.DeltaCheckpoint[source]¶
Last Delta version this producer published for a table, read back from an asset event.
- class airflow.providers.databricks.operators.delta_table.DatabricksDeltaTableVersionProducerOperator(*, unity_table, checkpoint_key=None, baseline_version=None, ignored_operations=None, max_history_rows=1000, on_recreation='publish', databricks_conn_id=DatabricksSqlHook.default_conn_name, http_path=None, sql_warehouse_name=None, session_configuration=None, http_headers=None, client_parameters=None, query_tags=None, include_airflow_query_tags=True, **kwargs)[source]¶
Bases:
airflow.providers.common.compat.sdk.BaseOperatorPublish an asset event when a Unity Catalog Delta table gets a newer committed version.
Use this operator in a recurring producer Dag to turn Delta commits made by other jobs or systems into Airflow asset events. Downstream Dags schedule on
unity_table.to_asset().Each run reads the newest Delta version, compares it with the checkpoint, and either publishes one asset event for the newest version or skips. Several commits between two runs are coalesced into one event for the newest version. Intermediate commits are not delivered one by one.
The checkpoint is the newest asset event that this producer published for the asset, found by
checkpoint_keyin the eventextra. Airflow records the event when the task succeeds, so publishing the event and advancing the checkpoint are the same write. Delivery is at least once: concurrent runs or a cleared run can publish the same version again. Consumers that must not process a version twice deduplicate onextra["dedup_id"].An event for version N does not pin downstream reads to N. A consumer that needs a reproducible read queries
VERSION AS OFextra["version"].See also
For more information on how to use this operator, take a look at the guide: DatabricksDeltaTableVersionProducerOperator
- Parameters:
unity_table (airflow.providers.databricks.assets.databricks.UnityTableIdentity) – Static identity of the observed table. It is added to
inlets(to read the checkpoint) and, whenoutletsis omitted,outletsis[unity_table.to_asset()]. Explicitoutletsmust includeunity_table.to_asset(), because the checkpoint is the event published on that asset.checkpoint_key (str | None) – Scope of the checkpoint. Producers with different keys keep separate checkpoints for the same asset. Defaults to
"<dag_id>.<task_id>". Changing it starts a new enrollment.baseline_version (int | None) – Version treated as already delivered when no checkpoint exists. When
None(the default), the first run publishes anenrollmentevent for the current version.ignored_operations (collections.abc.Collection[str] | None) – Delta history operations that do not count as a change. Defaults to
OPTIMIZE,VACUUM STARTandVACUUM END. Pass an empty collection to publish for every commit.max_history_rows (int) – Largest
DESCRIBE HISTORYwindow read to prove that all commits after the checkpoint are ignored operations. When the window or the retained history does not reach the checkpoint, the operator publishes rather than risk missing a data change.on_recreation (Literal['publish', 'fail']) – What to do when the table id differs from the checkpoint, or the version moved backwards, which means the table was dropped and recreated or replaced by a rename.
"publish"publishes arecreatedevent and checkpoints the new table."fail"raises and leaves the checkpoint untouched.databricks_conn_id (str) – Reference to Databricks connection id.
http_path (str | None) – HTTP path of a Databricks SQL warehouse or all-purpose cluster.
sql_warehouse_name (str | None) – Name of a Databricks SQL warehouse, used when
http_pathis not set.session_configuration (dict[str, str] | None) – Spark session parameters.
http_headers (list[tuple[str, str]] | None) – (k, v) pairs set as HTTP headers on every request.
client_parameters (dict[str, Any] | None) – Additional Databricks SQL connector parameters.
query_tags (dict[str, str | None] | None) – Query tags attached to the Databricks SQL queries. (templated)
include_airflow_query_tags (bool) – If True, add Airflow Dag/task/run metadata as query tags.
- template_fields: collections.abc.Sequence[str] = ('databricks_conn_id', 'http_headers', 'query_tags')[source]¶