airflow.providers.microsoft.azure.hooks.synapse¶

Attributes¶

Credentials

Exceptions¶

AzureSynapsePipelineRunException

An exception that indicates a pipeline run failed to complete.

Classes¶

`AzureSynapseSparkBatchRunStatus`	Azure Synapse Spark Job operation statuses.
`AzureSynapseHook`	A hook to interact with Azure Synapse.
`AzureSynapsePipelineRunStatus`	Azure Synapse pipeline operation statuses.
`BaseAzureSynapseHook`	A base hook class to create session and connection to Azure Synapse using connection id.
`AzureSynapsePipelineHook`	A hook to interact with Azure Synapse Pipeline.

Module Contents¶

airflow.providers.microsoft.azure.hooks.synapse.Credentials[source]¶

class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseSparkBatchRunStatus[source]¶

Azure Synapse Spark Job operation statuses.

NOT_STARTED = 'not_started'[source]¶

STARTING = 'starting'[source]¶

RUNNING = 'running'[source]¶

IDLE = 'idle'[source]¶

BUSY = 'busy'[source]¶

SHUTTING_DOWN = 'shutting_down'[source]¶

ERROR = 'error'[source]¶

DEAD = 'dead'[source]¶

KILLED = 'killed'[source]¶

SUCCESS = 'success'[source]¶

TERMINAL_STATUSES[source]¶

class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapseHook(azure_synapse_conn_id=default_conn_name, spark_pool='')[source]¶

Bases: airflow.hooks.base.BaseHook

A hook to interact with Azure Synapse.

Parameters:

azure_synapse_conn_id (str) – The Azure Synapse connection id.
spark_pool (str) – The Apache Spark pool used to submit the job

conn_type: str = 'azure_synapse'[source]¶

conn_name_attr: str = 'azure_synapse_conn_id'[source]¶

default_conn_name: str = 'azure_synapse_default'[source]¶

hook_name: str = 'Azure Synapse'[source]¶

classmethod get_connection_form_widgets()[source]¶

Return connection widgets to add to connection form.

classmethod get_ui_field_behaviour()[source]¶

Return custom field behaviour.

job_id: int | None = None[source]¶

conn_id = 'azure_synapse_default'[source]¶

spark_pool = ''[source]¶

get_conn()[source]¶

Return connection for the hook.

run_spark_job(payload)[source]¶

Run a job in an Apache Spark pool.

Parameters:: payload (azure.synapse.spark.models.SparkBatchJobOptions) – Livy compatible payload which represents the spark job that a user wants to submit.

get_job_run_status()[source]¶: Get the job run status.

wait_for_job_run_status(job_id, expected_statuses, check_interval=60, timeout=60 * 60 * 24 * 7)[source]¶

Wait for a job run to match an expected status.

Parameters:

job_id (int | None) – The job run identifier.
expected_statuses (str | set[str]) – The desired status(es) to check against a job run’s current status.
check_interval (int) – Time in seconds to check on a job run’s status.
timeout (int) – Time in seconds to wait for a job to reach a terminal status or the expected status.

cancel_job_run(job_id)[source]¶

Cancel the spark job run.

Parameters:: job_id (int) – The synapse spark job identifier.

class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineRunStatus[source]¶

Azure Synapse pipeline operation statuses.

QUEUED = 'Queued'[source]¶

IN_PROGRESS = 'InProgress'[source]¶

SUCCEEDED = 'Succeeded'[source]¶

FAILED = 'Failed'[source]¶

CANCELING = 'Canceling'[source]¶

CANCELLED = 'Cancelled'[source]¶

TERMINAL_STATUSES[source]¶

INTERMEDIATE_STATES[source]¶

FAILURE_STATES[source]¶

exception airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineRunException[source]¶

Bases: airflow.exceptions.AirflowException

An exception that indicates a pipeline run failed to complete.

class airflow.providers.microsoft.azure.hooks.synapse.BaseAzureSynapseHook(azure_synapse_conn_id=default_conn_name, **kwargs)[source]¶

Bases: airflow.hooks.base.BaseHook

A base hook class to create session and connection to Azure Synapse using connection id.

Parameters:: azure_synapse_conn_id (str) – The Azure Synapse connection id.

conn_type: str = 'azure_synapse'[source]¶

conn_name_attr: str = 'azure_synapse_conn_id'[source]¶

default_conn_name: str = 'azure_synapse_default'[source]¶

hook_name: str = 'Azure Synapse'[source]¶

classmethod get_connection_form_widgets()[source]¶

Return connection widgets to add to connection form.

classmethod get_ui_field_behaviour()[source]¶

Return custom field behaviour.

conn_id = 'azure_synapse_default'[source]¶

class airflow.providers.microsoft.azure.hooks.synapse.AzureSynapsePipelineHook(azure_synapse_workspace_dev_endpoint, azure_synapse_conn_id=default_conn_name, **kwargs)[source]¶

Bases: BaseAzureSynapseHook

A hook to interact with Azure Synapse Pipeline.

Parameters:

azure_synapse_conn_id (str) – The Azure Synapse connection id.
azure_synapse_workspace_dev_endpoint (str) – The Azure Synapse Workspace development endpoint.

default_conn_name: str = 'azure_synapse_connection'[source]¶

azure_synapse_workspace_dev_endpoint[source]¶

get_conn()[source]¶

Return connection for the hook.

run_pipeline(pipeline_name, **config)[source]¶

Run a Synapse pipeline.

Parameters:

pipeline_name (str) – The pipeline name.
config (Any) – Extra parameters for the Synapse Artifact Client.

Returns:

The pipeline run Id.

Return type:

azure.synapse.artifacts.models.CreateRunResponse

get_pipeline_run(run_id)[source]¶

Get the pipeline run.

Parameters:: run_id (str) – The pipeline run identifier.
Returns:: The pipeline run.
Return type:: azure.synapse.artifacts.models.PipelineRun

get_pipeline_run_status(run_id)[source]¶

Get a pipeline run’s current status.

Parameters:: run_id (str) – The pipeline run identifier.
Returns:: The status of the pipeline run.
Return type:: str

refresh_conn()[source]¶

wait_for_pipeline_run_status(run_id, expected_statuses, check_interval=60, timeout=60 * 60 * 24 * 7)[source]¶

Wait for a pipeline run to match an expected status.

Parameters:

run_id (str) – The pipeline run identifier.
expected_statuses (str | set[str]) – The desired status(es) to check against a pipeline run’s current status.
check_interval (int) – Time in seconds to check on a pipeline run’s status.
timeout (int) – Time in seconds to wait for a pipeline to reach a terminal status or the expected status.

Returns:

Boolean indicating if the pipeline run has reached the expected_status.

Return type:

bool

cancel_run_pipeline(run_id)[source]¶

Cancel the pipeline run.

Parameters:: run_id (str) – The pipeline run identifier.