airflow.providers.amazon.aws.executors.ecs.utils¶

AWS ECS Executor Utilities.

Data classes and utility functions used by the ECS executor.

Attributes¶

`CommandType`
`ExecutorConfigFunctionType`
`ExecutorConfigType`
`ECS_LAUNCH_TYPE_EC2`
`ECS_LAUNCH_TYPE_FARGATE`
`CONFIG_GROUP_NAME`
`CONFIG_DEFAULTS`

Exceptions¶

EcsExecutorException

Thrown when something unexpected has occurred within the ECS ecosystem.

Classes¶

`EcsQueuedTask`	Represents a queued ECS workload (task or callback). The workload will be run in the next heartbeat.
`EcsTaskInfo`	Contains information about a currently running ECS task.
`RunTaskKwargsConfigKeys`	Keys loaded into the config which are valid ECS run_task kwargs.
`AllEcsConfigKeys`	All keys loaded into the config which are related to the ECS Executor.
`EcsExecutorTask`	Data Transfer Object for an ECS Task.
`EcsTaskCollection`	A five-way dictionary between Airflow workload keys, commands, ECS ARNs, and ECS task objects.

Functions¶

`parse_assign_public_ip`(assign_public_ip[, ...])	Convert "assign_public_ip" from True/False to ENABLE/DISABLE.
`camelize_dict_keys`(nested_dict)	Accept a potentially nested dictionary and recursively convert all keys into camelCase.

Module Contents¶

type airflow.providers.amazon.aws.executors.ecs.utils.CommandType = Sequence[str] | Sequence[ExecuteTask] | Sequence[ExecuteCallback][source]¶

airflow.providers.amazon.aws.executors.ecs.utils.ExecutorConfigFunctionType[source]¶

airflow.providers.amazon.aws.executors.ecs.utils.ExecutorConfigType[source]¶

airflow.providers.amazon.aws.executors.ecs.utils.ECS_LAUNCH_TYPE_EC2 = 'EC2'[source]¶

airflow.providers.amazon.aws.executors.ecs.utils.ECS_LAUNCH_TYPE_FARGATE = 'FARGATE'[source]¶

airflow.providers.amazon.aws.executors.ecs.utils.CONFIG_GROUP_NAME = 'aws_ecs_executor'[source]¶

airflow.providers.amazon.aws.executors.ecs.utils.CONFIG_DEFAULTS[source]¶

class airflow.providers.amazon.aws.executors.ecs.utils.EcsQueuedTask[source]¶

Represents a queued ECS workload (task or callback). The workload will be run in the next heartbeat.

key: airflow.executors.workloads.types.WorkloadKey[source]¶

command: CommandType[source]¶

queue: str | None[source]¶

executor_config: ExecutorConfigType[source]¶

attempt_number: int[source]¶

next_attempt_time: datetime.datetime[source]¶

class airflow.providers.amazon.aws.executors.ecs.utils.EcsTaskInfo[source]¶

Contains information about a currently running ECS task.

cmd: CommandType[source]¶

queue: str | None[source]¶

config: ExecutorConfigType[source]¶

class airflow.providers.amazon.aws.executors.ecs.utils.RunTaskKwargsConfigKeys[source]¶

Bases: airflow.providers.amazon.aws.executors.utils.base_config_keys.BaseConfigKeys

Keys loaded into the config which are valid ECS run_task kwargs.

ASSIGN_PUBLIC_IP = 'assign_public_ip'[source]¶

CAPACITY_PROVIDER_STRATEGY = 'capacity_provider_strategy'[source]¶

CLUSTER = 'cluster'[source]¶

CONTAINER_NAME = 'container_name'[source]¶

LAUNCH_TYPE = 'launch_type'[source]¶

PLATFORM_VERSION = 'platform_version'[source]¶

SECURITY_GROUPS = 'security_groups'[source]¶

SUBNETS = 'subnets'[source]¶

TASK_DEFINITION = 'task_definition'[source]¶

class airflow.providers.amazon.aws.executors.ecs.utils.AllEcsConfigKeys[source]¶

Bases: RunTaskKwargsConfigKeys

All keys loaded into the config which are related to the ECS Executor.

AWS_CONN_ID = 'conn_id'[source]¶

CHECK_HEALTH_ON_STARTUP = 'check_health_on_startup'[source]¶

MAX_RUN_TASK_ATTEMPTS = 'max_run_task_attempts'[source]¶

REGION_NAME = 'region_name'[source]¶

RUN_TASK_KWARGS = 'run_task_kwargs'[source]¶

exception airflow.providers.amazon.aws.executors.ecs.utils.EcsExecutorException[source]¶

Bases: Exception

Thrown when something unexpected has occurred within the ECS ecosystem.

class airflow.providers.amazon.aws.executors.ecs.utils.EcsExecutorTask(task_arn, last_status, desired_status, containers, started_at=None, stopped_reason=None, external_executor_id=None)[source]¶

Data Transfer Object for an ECS Task.

task_arn[source]¶

last_status[source]¶

desired_status[source]¶

containers[source]¶

started_at = None[source]¶

stopped_reason = None[source]¶

external_executor_id = None[source]¶

get_task_state()[source]¶

Determine the state of an ECS task based on its status and other relevant attributes.

It can return one of the following statuses:: QUEUED - Task is being provisioned. RUNNING - Task is launched on ECS. REMOVED - Task provisioning has failed for some reason. See stopped_reason. FAILED - Task is completed and at least one container has failed. SUCCESS - Task is completed and all containers have succeeded.

__repr__()[source]¶: Return a string representation of the ECS task.

class airflow.providers.amazon.aws.executors.ecs.utils.EcsTaskCollection[source]¶

A five-way dictionary between Airflow workload keys, commands, ECS ARNs, and ECS task objects.

key_to_arn: dict[airflow.executors.workloads.types.WorkloadKey, str][source]¶

arn_to_key: dict[str, airflow.executors.workloads.types.WorkloadKey][source]¶

tasks: dict[str, EcsExecutorTask][source]¶

key_to_failure_counts: dict[airflow.executors.workloads.types.WorkloadKey, int][source]¶

key_to_task_info: dict[airflow.executors.workloads.types.WorkloadKey, EcsTaskInfo][source]¶

add_task(task, airflow_task_key, queue, airflow_cmd, exec_config, attempt_number)[source]¶

Add a task to the collection.

update_task(task)[source]¶

Update the state of the given task based on task ARN.

task_by_key(task_key)[source]¶

Get a task by Airflow workload key.

task_by_arn(arn)[source]¶

Get a task by AWS ARN.

pop_by_key(task_key)[source]¶

Delete task from collection based off of Airflow workload key.

get_all_arns()[source]¶

Get all AWS ARNs in collection.

get_all_task_keys()[source]¶

Get all Airflow workload keys in collection.

failure_count_by_key(task_key)[source]¶

Get the number of times a workload has failed given an Airflow workload key.

increment_failure_count(task_key)[source]¶

Increment the failure counter given an Airflow workload key.

info_by_key(task_key)[source]¶

Get the task info given an Airflow workload key.

__getitem__(value)[source]¶: Get a task by AWS ARN.

__len__()[source]¶: Determine the number of tasks in collection.

airflow.providers.amazon.aws.executors.ecs.utils.parse_assign_public_ip(assign_public_ip, is_launch_type_ec2=False)[source]¶: Convert “assign_public_ip” from True/False to ENABLE/DISABLE.

airflow.providers.amazon.aws.executors.ecs.utils.camelize_dict_keys(nested_dict)[source]¶

Accept a potentially nested dictionary and recursively convert all keys into camelCase.