Databricks SQL warehouse lifecycle operators¶

Use DatabricksStartWarehouseOperator and DatabricksStopWarehouseOperator to start and stop an existing Databricks SQL warehouse through the Databricks SQL Warehouses API. Both operators require the warehouse ID and use the Databricks connection for authentication.

By default, each operator waits for the requested state: RUNNING when starting and STOPPED when stopping. Use polling_period_seconds to control the polling interval, timeout to limit the wait, or wait_for_termination=False to return after requesting the transition. Set deferrable=True (or enable [operators] default_deferrable) to wait on the triggerer instead of holding a worker slot. In deferrable mode the operator records an absolute end_time from timeout when it defers, so a triggerer restart does not reset the wait. wait_for_termination=False still returns immediately and does not defer. Clearing a deferred warehouse wait does not start or stop the warehouse: Databricks has no cancel API for these transitions, and a cancelled start must not stop a warehouse the Dag still needs.

Repeated task attempts are safe: an already running warehouse is not started again, and an already stopped warehouse is not stopped again. If a start is requested while a warehouse is stopping, any transition rejection from Databricks is propagated to the task.

Start a SQL warehouse¶

tests/system/databricks/example_databricks_sql_warehouse.py[source]

    start_warehouse = DatabricksStartWarehouseOperator(
        task_id="start_warehouse",
        databricks_conn_id="databricks_default",
        warehouse_id=WAREHOUSE_ID,
        wait_for_termination=True,
    )

Stop a SQL warehouse¶

The stop task uses the all_done trigger rule so the warehouse is stopped even when an upstream task fails.

tests/system/databricks/example_databricks_sql_warehouse.py[source]

    stop_warehouse = DatabricksStopWarehouseOperator(
        task_id="stop_warehouse",
        databricks_conn_id="databricks_default",
        warehouse_id=WAREHOUSE_ID,
        wait_for_termination=True,
        trigger_rule="all_done",
    )