airflow.providers.amazon.aws.executors.aws_lambda.lambda_executor¶

Attributes¶

WorkloadKey

Classes¶

AwsLambdaExecutor

An Airflow Executor that submits workloads (tasks and callbacks) to AWS Lambda asynchronously.

Module Contents¶

type airflow.providers.amazon.aws.executors.aws_lambda.lambda_executor.WorkloadKey = _WorkloadKey[source]¶

class airflow.providers.amazon.aws.executors.aws_lambda.lambda_executor.AwsLambdaExecutor(*args, **kwargs)[source]¶

Bases: airflow.executors.base_executor.BaseExecutor

An Airflow Executor that submits workloads (tasks and callbacks) to AWS Lambda asynchronously.

When execute_async() is called, the executor invokes a specified AWS Lambda function (asynchronously) with a payload that includes the workload command and a unique workload key.

The Lambda function writes its result directly to an SQS queue, which is then polled by this executor to update workload state in Airflow.

supports_multi_team: bool = True[source]¶

supports_callbacks: bool = True[source]¶

queued_tasks: dict[WorkloadKey, airflow.executors.workloads.All][source]¶

pending_workloads: collections.deque[source]¶

running_workloads: dict[str, WorkloadKey][source]¶

lambda_function_name[source]¶

sqs_queue_url[source]¶

dlq_url[source]¶

qualifier[source]¶

max_invoke_attempts[source]¶

attempts_since_last_successful_connection = 0[source]¶

IS_BOTO_CONNECTION_HEALTHY = False[source]¶

property pending_tasks: collections.deque[source]¶

Deprecated: use pending_workloads.

property running_tasks: dict[str, WorkloadKey][source]¶

Deprecated: use running_workloads.

start()[source]¶: Call this when the Executor is run for the first time by the scheduler.

check_health()[source]¶

Check the health of the Lambda and SQS connections.

For lambda: Use get_function to test if the lambda connection works and the function can be described. For SQS: Use get_queue_attributes is used as a close analog to describe to test if the SQS connection is working.

load_connections(check_connection=True)[source]¶

Retrieve the AWS connection via Hooks to leverage the Airflow connection system.

Parameters:: check_connection (bool) – If True, check the health of the connection after loading it.

sync()[source]¶

Sync the executor with the current state of workloads.

Check in on currently running tasks and callbacks and attempt to run any new workloads that have been queued.

queue_workload(workload, session)[source]¶

execute_async(key, command, queue=None, executor_config=None)[source]¶

Save the workload to be executed in the next sync by inserting the commands into a queue.

Parameters:

key (WorkloadKey) – Unique workload key. Task workloads use TaskInstanceKey, callback workloads use a string id.
command (airflow.providers.amazon.aws.executors.aws_lambda.utils.CommandType) – The workload command or serialized shell command to execute.
executor_config – (Unused) to keep the same signature as the base.
queue – (Unused) to keep the same signature as the base.

attempt_workload_runs()[source]¶

Attempt to run workloads that are queued in the pending_workloads.

Each workload is submitted to AWS Lambda with a payload containing the workload key and command. The workload key is used to track the workload’s state in Airflow.

attempt_task_runs()[source]¶: Use attempt_workload_runs as attempt_task_runs is deprecated.

sync_running_workloads()[source]¶

Poll the SQS queue for messages indicating workload completion.

Each message is expected to contain a JSON payload with ‘task_key’ and ‘return_code’. Based on the return code, update the workload state accordingly.

sync_running_tasks()[source]¶: Use sync_running_workloads as sync_running_tasks is deprecated.

process_queue(queue_url)[source]¶

Poll the SQS queue for messages indicating workload completion.

Each message is expected to contain a JSON payload with ‘task_key’ and ‘return_code’.

Based on the return code, update the workload state accordingly.

try_adopt_task_instances(tis)[source]¶

Adopt task instances which have an external_executor_id (the serialized workload key).

The external_executor_id represents the workload identifier. In legacy executors (Airflow < 3.3) this is the serialized TaskInstanceKey. In the workload-based executor model (Airflow ≥ 3.3) this corresponds to the WorkloadKey.

Anything that is not adopted will be cleared by the scheduler and becomes eligible for re-scheduling.

Parameters:: tis (collections.abc.Sequence[airflow.models.taskinstance.TaskInstance]) – The task instances to adopt.

end(heartbeat_interval=10)[source]¶

End execution. Poll until all outstanding workloads are marked as completed.

This is a blocking call and async Lambda workloads cannot be cancelled, so this will wait until all workloads are either completed or the timeout is reached.

Parameters:: heartbeat_interval – The interval in seconds to wait between checks for workload completion.

terminate()[source]¶: Get called when the daemon receives a SIGTERM.