tests.system.databricks.example_databricks_delta_producer

Example Dags that turn Delta table commits made outside Airflow into asset events.

The producer Dag polls a Unity Catalog table every 10 minutes and publishes an asset event when the table has a newer Delta version. The consumer Dag is scheduled on that asset and reads the exact version that the event reports.

Attributes

ENV_ID

DAG_ID

DATABRICKS_HOST

SQL_WAREHOUSE_NAME

orders

test_run

Module Contents

tests.system.databricks.example_databricks_delta_producer.ENV_ID[source]
tests.system.databricks.example_databricks_delta_producer.DAG_ID = 'example_databricks_delta_producer'[source]
tests.system.databricks.example_databricks_delta_producer.DATABRICKS_HOST[source]
tests.system.databricks.example_databricks_delta_producer.SQL_WAREHOUSE_NAME[source]
tests.system.databricks.example_databricks_delta_producer.orders[source]
tests.system.databricks.example_databricks_delta_producer.test_run[source]

Was this entry helpful?