Configuration Reference¶

This page contains the list of all available Airflow configurations for the apache-airflow-providers-amazon provider that can be set in the airflow.cfg file or using environment variables.

Note

The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.

Note

For more information see Setting Configuration Options.

[aws]¶

This section contains settings for Amazon Web Services (AWS) integration.

cloudwatch_task_handler_json_serializer¶

Added in version 8.7.2.

By default, when logging non-string messages, all non-json objects are logged as null. Except datetime objects which are ISO formatted. Users can optionally use a repr serializer or provide their own JSON serializer for any non-JSON-serializable objects in the logged message.

airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize uses repr (be aware there is the potential of logging sensitive data depending on the repr method of logged objects)
airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize_legacy uses null.

If a custom serializer is provided, it must adhere to Callable[[Any], str | None], where None serializes to null (e.g. def my_serializer(o: Any) -> str | None). Since this is on the logging path and it’s possible there’s an exception being handled, special care should be taken to fail gracefully without raising a new exception inside of your serializer.

Type:: string
Default:: airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize_legacy
Environment Variable:: AIRFLOW__AWS__CLOUDWATCH_TASK_HANDLER_JSON_SERIALIZER
Example:: airflow.providers.amazon.aws.log.cloudwatch_task_handler.json_serialize

session_factory¶

Added in version 3.1.1.

Full import path to the class which implements a custom session factory for boto3.session.Session. For more details please have a look at Session Factory.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS__SESSION_FACTORY
Example:: my_company.aws.MyCustomSessionFactory

[aws_auth_manager]¶

This section only applies if you are using the AwsAuthManager. In other words, if you set [core] auth_manager = airflow.providers.amazon.aws.auth_manager.aws_auth_manager.AwsAuthManager in Airflow’s configuration.

avp_policy_store_id¶

Added in version 8.12.0.

Amazon Verified Permissions’ policy store ID where all the policies defining user permissions in Airflow are stored. Required.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_AUTH_MANAGER__AVP_POLICY_STORE_ID

conn_id¶

Added in version 8.12.0.

The Airflow connection (i.e. credentials) used by the AWS auth manager to make API calls to AWS Identity Center and Amazon Verified Permissions.

Type:: string
Default:: aws_default
Environment Variable:: AIRFLOW__AWS_AUTH_MANAGER__CONN_ID
Example:: aws_default

enable¶

Added in version 8.12.0.

AWS auth manager is not ready to be used. Turn on this flag to use it anyway. Do that at your own risk since the AWS auth manager is not in an usable state.

Type:: boolean
Default:: False
Environment Variable:: AIRFLOW__AWS_AUTH_MANAGER__ENABLE
Example:: True

region_name¶

Added in version 8.10.

The name of the AWS Region where Amazon Verified Permissions is configured. Required.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_AUTH_MANAGER__REGION_NAME
Example:: us-east-1

saml_metadata_url¶

Added in version 8.12.0.

SAML metadata XML file provided by AWS Identity Center. This URL can be found in the AWS Identity Center console. Required.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_AUTH_MANAGER__SAML_METADATA_URL
Example:: https://portal.sso.<region>.amazonaws.com/saml/metadata/XXXXXXXXXX

[aws_batch_executor]¶

This section only applies if you are using the AwsBatchExecutor in Airflow’s [core] configuration. For more information on any of these execution parameters, see the link below: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch.html#Batch.Client.submit_job For boto3 credential management, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

check_health_on_startup¶

Added in version 8.11.

Whether or not to check the Batch Executor health on startup.

Type:: boolean
Default:: True
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__CHECK_HEALTH_ON_STARTUP
Example:: True

conn_id¶

Added in version 8.11.

The Airflow connection (i.e. credentials) used by the Batch executor to make API calls to AWS Batch.

Type:: string
Default:: aws_default
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__CONN_ID
Example:: aws_default

job_definition¶

Added in version 8.11.

The job definition used by the job. You can specify either the name or the Amazon Resource Name (ARN) of the job definition with or without the revision. If the revision is not specified, then the latest active revision is used.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__JOB_DEFINITION
Example:: airflow-batch-executor-job-definition

job_name¶

Added in version 8.11.

The name of the job submitted to AWS Batch. It can be up to 128 letters long. The first character must be alphanumeric, can contain uppercase and lowercase letters, numbers, hyphens (-), and underscores (_).

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__JOB_NAME
Example:: airflow-batch-executor-job

job_queue¶

Added in version 8.11.

The job queue where the job is submitted. You can specify either the name or the Amazon Resource Name (ARN) of the queue.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__JOB_QUEUE
Example:: airflow-batch-executor-job-queue

max_submit_job_attempts¶

Added in version 8.11.

The maximum number of times the Batch Executor should attempt to run a Batch Job.

Type:: integer
Default:: 3
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__MAX_SUBMIT_JOB_ATTEMPTS
Example:: 3

region_name¶

Added in version 8.11.

The name of the AWS Region where Amazon Batch is configured. Required.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__REGION_NAME
Example:: us-east-1

submit_job_kwargs¶

Added in version 8.11.

Additional parameters to pass to the submit_job method of the AWS Batch client.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_BATCH_EXECUTOR__SUBMIT_JOB_KWARGS
Example:: {"Tags": [{"Key": "key", "Value": "value"}]}

[aws_ecs_executor]¶

This section only applies if you are using the AwsEcsExecutor in Airflow’s [core] configuration. For more information on any of these execution parameters, see the link below: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs/client/run_task.html For boto3 credential management, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

assign_public_ip¶

Added in version 8.10.

Whether to assign a public IP address to the containers launched by the ECS executor. For more info see url to Boto3 docs above.

Type:: boolean
Default:: False
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__ASSIGN_PUBLIC_IP
Example:: True

capacity_provider_strategy¶

Added in version 8.17.

The capacity provider strategy to use for the task.

If a Capacity Provider Strategy is specified, the Launch Type parameter must be omitted. If no Capacity Provider Strategy or Launch Type is specified, the Default CapacityProvider Strategy for the cluster is used, if present.

When you use cluster auto scaling, you must specify Capacity Provider Strategy and not Launch Type.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__CAPACITY_PROVIDER_STRATEGY
Example:: [{'capacityProvider': 'cp1', 'weight': 5}, {'capacityProvider': 'cp2', 'weight': 1}]

check_health_on_startup¶

Added in version 8.11.

Whether or not to check the ECS Executor health on startup.

Type:: boolean
Default:: True
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__CHECK_HEALTH_ON_STARTUP
Example:: True

cluster¶

Added in version 8.10.

Name of the Amazon ECS Cluster. Required.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__CLUSTER
Example:: ecs_executor_cluster

conn_id¶

Added in version 8.10.

The Airflow connection (i.e. credentials) used by the ECS executor to make API calls to AWS ECS.

Type:: string
Default:: aws_default
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__CONN_ID
Example:: aws_default

container_name¶

Added in version 8.10.

Name of the container that will be used to execute Airflow tasks via the ECS executor. The container should be specified in the ECS Task Definition and will receive an airflow CLI command as an additional parameter to its entrypoint. For more info see url to Boto3 docs above. Required.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__CONTAINER_NAME
Example:: ecs_executor_container

launch_type¶

Added in version 8.10.

Launch type can either be ‘FARGATE’ OR ‘EC2’. For more info see url to Boto3 docs above.

If a Launch Type is specified, the Capacity Provider Strategy parameter must be omitted. If no Capacity Provider Strategy or Launch Type is specified, the Default Capacity Provider Strategy for the cluster is used, if present.

If the launch type is EC2, the executor will attempt to place tasks on empty EC2 instances. If there are no EC2 instances available, no task is placed and this function will be called again in the next heart-beat.

If the launch type is FARGATE, this will run the tasks on new AWS Fargate instances.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__LAUNCH_TYPE
Example:: FARGATE

max_run_task_attempts¶

Added in version 8.10.

The maximum number of times the ECS Executor should attempt to run a task.

Type:: integer
Default:: 3
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__MAX_RUN_TASK_ATTEMPTS
Example:: 3

platform_version¶

Added in version 8.10.

The platform version the task uses. A platform version is only specified for tasks hosted on Fargate. If one isn’t specified, the LATEST platform version is used.

Type:: string
Default:: LATEST
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__PLATFORM_VERSION
Example:: 1.4.0

region_name¶

Added in version 8.10.

The name of the AWS Region where Amazon ECS is configured. Required.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__REGION_NAME
Example:: us-east-1

run_task_kwargs¶

Added in version 8.10.

A JSON string containing arguments to provide the ECS run_task API (see url above).

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__RUN_TASK_KWARGS
Example:: {"tags": {"key": "schema", "value": "1.0"}}

security_groups¶

Added in version 8.10.

The comma-separated IDs of the security groups associated with the task. If you don’t specify a security group, the default security group for the VPC is used. There’s a limit of 5 security groups. For more info see url to Boto3 docs above.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__SECURITY_GROUPS
Example:: sg-XXXX,sg-YYYY

subnets¶

Added in version 8.10.

The comma-separated IDs of the subnets associated with the task or service. There’s a limit of 16 subnets. For more info see url to Boto3 docs above.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__SUBNETS
Example:: subnet-XXXXXXXX,subnet-YYYYYYYY

task_definition¶

Added in version 8.10.

The family and revision (family:revision) or full ARN of the task definition to run. If a revision isn’t specified, the latest ACTIVE revision is used. For more info see url to Boto3 docs above.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__AWS_ECS_EXECUTOR__TASK_DEFINITION
Example:: executor_task_definition:LATEST