Azure Data Factory Operators¶
Azure Data Factory is Azure’s cloud ETL service for scale-out serverless data integration and data transformation. It offers a code-free UI for intuitive authoring and single-pane-of-glass monitoring and management.
AzureDataFactoryRunPipelineOperator¶
Use the AzureDataFactoryRunPipelineOperator to execute a pipeline within a data factory.
By default, the operator will periodically check on the status of the executed pipeline to terminate with a “Succeeded” status.
This functionality can be disabled for an asynchronous wait – typically with the AzureDataFactoryPipelineRunStatusSensor – by setting wait_for_termination to False.
Below is an example of using this operator to execute an Azure Data Factory pipeline.
run_pipeline1 = AzureDataFactoryRunPipelineOperator( task_id="run_pipeline1", pipeline_name="pipeline1", parameters={"myParam": "value"}, )
Below is an example of using this operator to execute an Azure Data Factory pipeline with a deferrable flag so that polling for the status of the pipeline run occurs on the Airflow Triggerer.
run_pipeline3 = AzureDataFactoryRunPipelineOperator( task_id="run_pipeline3", pipeline_name="pipeline1", parameters={"myParam": "value"}, deferrable=True, )
Here is a different example of using this operator to execute a pipeline but coupled with the AzureDataFactoryPipelineRunStatusSensor to perform an asynchronous wait.
run_pipeline2 = AzureDataFactoryRunPipelineOperator( task_id="run_pipeline2", pipeline_name="pipeline2", wait_for_termination=False, ) pipeline_run_sensor = AzureDataFactoryPipelineRunStatusSensor( task_id="pipeline_run_sensor", run_id=cast("str", XComArg(run_pipeline2, key="run_id")), ) # Performs polling on the Airflow Triggerer thus freeing up resources on Airflow Worker pipeline_run_sensor_deferred = AzureDataFactoryPipelineRunStatusSensor( task_id="pipeline_run_sensor_defered", run_id=cast("str", XComArg(run_pipeline2, key="run_id")), deferrable=True, ) pipeline_run_async_sensor = AzureDataFactoryPipelineRunStatusSensor( task_id="pipeline_run_async_sensor", run_id=cast("str", XComArg(run_pipeline2, key="run_id")), deferrable=True, )
Also you can use deferrable mode in AzureDataFactoryPipelineRunStatusSensor if you would like to free up the worker slots while the sensor is running.
run_pipeline2 = AzureDataFactoryRunPipelineOperator( task_id="run_pipeline2", pipeline_name="pipeline2", wait_for_termination=False, ) pipeline_run_sensor = AzureDataFactoryPipelineRunStatusSensor( task_id="pipeline_run_sensor", run_id=cast("str", XComArg(run_pipeline2, key="run_id")), ) # Performs polling on the Airflow Triggerer thus freeing up resources on Airflow Worker pipeline_run_sensor_deferred = AzureDataFactoryPipelineRunStatusSensor( task_id="pipeline_run_sensor_defered", run_id=cast("str", XComArg(run_pipeline2, key="run_id")), deferrable=True, ) pipeline_run_async_sensor = AzureDataFactoryPipelineRunStatusSensor( task_id="pipeline_run_async_sensor", run_id=cast("str", XComArg(run_pipeline2, key="run_id")), deferrable=True, )
Durable execution¶
The operator is durable by default: it keeps the
pipeline run id in the task state store before it
waits. A retry reconnects to a run that is still in progress, in both regular and deferrable mode, and does
not start a new one once the run has succeeded. A run that failed, was cancelled, is being cancelled or no
longer exists is started again. With wait_for_termination=False, a retry returns without waiting when the
stored run is still in progress. In deferrable mode, an error while polling no longer cancels the run, so
the retry can reconnect to it. Durable execution requires Airflow 3.3 or newer.
Clearing the task instance from the UI or the REST API discards the stored run id by default, so the next
try starts a new pipeline run. To keep the stored run id instead, tick “keep task state” in the clear
dialog or pass keep_task_state=true to the clear endpoint. The next try then treats the stored run
like a retry does.
Before Airflow 3.5, clearing the whole Dag run, marking a task as success or failed, and
airflow tasks clear / airflow dags clear keep the stored run id. So if you clear the whole Dag run,
a task whose pipeline run succeeded returns without running the pipeline again. Clear the task instance
itself instead. From Airflow 3.5, every way of clearing discards the stored run id by default.
Clearing a task that is waiting on the worker cancels its pipeline run first, so the next try starts a new run either way. Only a deferred task’s run keeps going. See Resumable Tasks for details.
Set durable=False to start a new pipeline run on every try, and to cancel the run when polling fails in
deferrable mode. Below Airflow 3.3 the operator behaves as with durable=False.
Poll for status of a data factory pipeline run asynchronously¶
Use the AzureDataFactoryPipelineRunStatusAsyncSensor
(deferrable version) to periodically retrieve the
status of a data factory pipeline run asynchronously. This sensor will free up the worker slots since
polling for job status happens on the Airflow triggerer, leading to efficient utilization
of resources within Airflow.
run_pipeline2 = AzureDataFactoryRunPipelineOperator(
task_id="run_pipeline2",
pipeline_name="pipeline2",
wait_for_termination=False,
)
pipeline_run_sensor = AzureDataFactoryPipelineRunStatusSensor(
task_id="pipeline_run_sensor",
run_id=cast("str", XComArg(run_pipeline2, key="run_id")),
)
# Performs polling on the Airflow Triggerer thus freeing up resources on Airflow Worker
pipeline_run_sensor_deferred = AzureDataFactoryPipelineRunStatusSensor(
task_id="pipeline_run_sensor_defered",
run_id=cast("str", XComArg(run_pipeline2, key="run_id")),
deferrable=True,
)
pipeline_run_async_sensor = AzureDataFactoryPipelineRunStatusSensor(
task_id="pipeline_run_async_sensor",
run_id=cast("str", XComArg(run_pipeline2, key="run_id")),
deferrable=True,
)
Reference¶
For further information, please refer to the Microsoft documentation: