`airflow.providers.databricks.plugins.databricks_workflow`¶

Module Contents¶

Classes¶

`WorkflowJobRunLink`	Constructs a link to monitor a Databricks Job Run.
`WorkflowJobRepairAllFailedLink`	Constructs a link to send a request to repair all failed tasks in the Databricks workflow.
`WorkflowJobRepairSingleTaskLink`	Construct a link to send a repair request for a single databricks task.
`RepairDatabricksTasks`	Repair databricks tasks from Airflow.
`DatabricksWorkflowPlugin`	Databricks Workflows plugin for Airflow.

Functions¶

`get_auth_decorator`()
`get_databricks_task_ids`(group_id, task_map, log)	Return a list of all Databricks task IDs for a dictionary of Airflow tasks.
`get_launch_task_id`(task_group)	Retrieve the launch task ID from the current task group or a parent task group, recursively.
`get_task_instance`(operator, dttm[, session])
`get_xcom_result`(ti_key, key)

Attributes¶

`REPAIR_WAIT_ATTEMPTS`
`REPAIR_WAIT_DELAY`
`airflow_app`
`repair_databricks_view`
`repair_databricks_package`

airflow.providers.databricks.plugins.databricks_workflow.REPAIR_WAIT_ATTEMPTS[source]¶

airflow.providers.databricks.plugins.databricks_workflow.REPAIR_WAIT_DELAY[source]¶

airflow.providers.databricks.plugins.databricks_workflow.airflow_app[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_auth_decorator()[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_databricks_task_ids(group_id, task_map, log)[source]¶

Return a list of all Databricks task IDs for a dictionary of Airflow tasks.

Parameters

group_id (str) – The task group ID.
task_map (dict[str, airflow.models.BaseOperator]) – A dictionary mapping task IDs to BaseOperator instances.
log (logging.Logger) – The logger to use for logging.

Returns

A list of Databricks task IDs for the given task group.

Return type

list[str]

airflow.providers.databricks.plugins.databricks_workflow.get_launch_task_id(task_group)[source]¶

Retrieve the launch task ID from the current task group or a parent task group, recursively.

Parameters: task_group (airflow.utils.task_group.TaskGroup) – Task Group to be inspected
Returns: launch Task ID
Return type: str

airflow.providers.databricks.plugins.databricks_workflow.get_task_instance(operator, dttm, session=NEW_SESSION)[source]¶

airflow.providers.databricks.plugins.databricks_workflow.get_xcom_result(ti_key, key)[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRunLink(context=None)[source]¶

Bases: airflow.models.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Constructs a link to monitor a Databricks Job Run.

name = 'See Databricks Job Run'[source]¶

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Note: The old signature of this function was (self, operator, dttm: datetime). That is still supported at runtime but is deprecated.

Parameters

operator (airflow.models.BaseOperator) – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns

link to external system

Return type

str

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRepairAllFailedLink(context=None)[source]¶

Bases: airflow.models.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Constructs a link to send a request to repair all failed tasks in the Databricks workflow.

name = 'Repair All Failed Tasks'[source]¶

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Note: The old signature of this function was (self, operator, dttm: datetime). That is still supported at runtime but is deprecated.

Parameters

operator – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns

link to external system

Return type

str

classmethod get_task_group_children(task_group)[source]¶

Given a TaskGroup, return children which are Tasks, inspecting recursively any TaskGroups within.

Parameters: task_group (airflow.utils.task_group.TaskGroup) – An Airflow TaskGroup
Returns: Dictionary that contains Task IDs as keys and Tasks as values.
Return type: dict[str, airflow.models.BaseOperator]

get_tasks_to_run(ti_key, operator, log)[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.WorkflowJobRepairSingleTaskLink(context=None)[source]¶

Bases: airflow.models.BaseOperatorLink, airflow.utils.log.logging_mixin.LoggingMixin

Construct a link to send a repair request for a single databricks task.

name = 'Repair a single task'[source]¶

get_link(operator, dttm=None, *, ti_key=None)[source]¶

Link to external system.

Note: The old signature of this function was (self, operator, dttm: datetime). That is still supported at runtime but is deprecated.

Parameters

operator – The Airflow operator object this link is associated to.
ti_key (airflow.models.taskinstance.TaskInstanceKey | None) – TaskInstance ID to return link for.

Returns

link to external system

Return type

str

class airflow.providers.databricks.plugins.databricks_workflow.RepairDatabricksTasks[source]¶

Bases: airflow.www.views.AirflowBaseView, airflow.utils.log.logging_mixin.LoggingMixin

Repair databricks tasks from Airflow.

default_view = 'repair'[source]¶

repair(dag_id, run_id)[source]¶

airflow.providers.databricks.plugins.databricks_workflow.repair_databricks_view[source]¶

airflow.providers.databricks.plugins.databricks_workflow.repair_databricks_package[source]¶

class airflow.providers.databricks.plugins.databricks_workflow.DatabricksWorkflowPlugin[source]¶

Bases: airflow.plugins_manager.AirflowPlugin

Databricks Workflows plugin for Airflow.

airflow.providers.databricks.plugins.databricks_workflow¶

Module Contents¶

Classes¶

Functions¶

Attributes¶

`airflow.providers.databricks.plugins.databricks_workflow`¶