AWS EKS Executor

The EKS executor runs each Airflow task in its own pod on an Amazon EKS cluster.

It extends the Kubernetes executor from the cncf.kubernetes provider, so pod scheduling, pod templates, per-task pod overrides, log handling and requirements such as a database backend other than SQLite are all as described in Kubernetes Executor. What it adds is authentication: it builds the Kubernetes client for your EKS cluster from an Airflow AWS connection, so the scheduler needs no kubeconfig file and you do not have to refresh cluster credentials yourself.

For a quick start guide please see here.

Requirements

The executor needs apache-airflow-providers-cncf-kubernetes version 10.24.0 or newer, the release that added the client_factory setting the executor is built on. The executor checks this when it is imported and raises a clear error if an older version is installed. Install the cncf.kubernetes extra of the Amazon provider together with that version:

pip install 'apache-airflow-providers-amazon[cncf.kubernetes]' \
    'apache-airflow-providers-cncf-kubernetes>=10.24.0'

The executor also relies on botocore using the regional STS endpoint by default (botocore 1.40 and newer), and the Amazon provider already requires botocore>=1.42.79, so no extra install step is needed.

The AWS credentials the executor uses must be allowed to call eks:DescribeCluster on the cluster, and their IAM principal must also be granted access inside the cluster, as described in Grant access to the cluster. Without that in-cluster grant every Kubernetes API call is rejected.

How authentication works

When the executor starts, it plugs its own Kubernetes client into the Kubernetes executor. Every process that needs a client, including the separate pod watcher process, builds its own from the [aws_eks_executor] settings: it describes the cluster to find its API endpoint and certificate authority, then gets an authentication token for it. An EKS token is a presigned STS URL that stays valid for roughly fifteen minutes, so the executor registers a refresh hook that the Kubernetes client calls on every authenticated request. The hook reuses the current token and gets a new one a minute before it expires.

Before it accepts any tasks, the executor checks that the cluster is ACTIVE or UPDATING, since EKS keeps the Kubernetes API available during an update (see Update existing cluster to new Kubernetes version). Unless check_health_on_startup is turned off, it also checks that it is allowed to list pods in its namespace. It refuses to start if either check fails.

Config Options

The executor reads its own settings from an aws_eks_executor section in airflow.cfg, or from environment variables of the form AIRFLOW__AWS_EKS_EXECUTOR__<OPTION_NAME>, for example AIRFLOW__AWS_EKS_EXECUTOR__CLUSTER_NAME=airflow-eks-cluster. For more information on how to set these options, see Setting Configuration Options.

Required config options:

  • CLUSTER_NAME - The name of the Amazon EKS cluster that tasks run on. The executor refuses to start if this is unset.

Optional config options:

  • CONN_ID - The Airflow connection (i.e. credentials) used by the EKS executor to make API calls to Amazon EKS. Defaults to aws_default.

  • REGION_NAME - The AWS Region the cluster is in. When this is left empty, it falls back to the AWS connection’s region, then boto3’s.

  • CHECK_HEALTH_ON_STARTUP - Whether to check on startup that the executor can list pods in its namespace. Defaults to True.

Pod-level configuration

Settings for the worker pods stay in the [kubernetes_executor] section, exactly as for the Kubernetes executor. This includes namespace, pod_template_file, worker_container_repository, worker_container_tag, and delete_worker_pods. See Kubernetes Executor for the full list, and for how pod templates and pod_override work.

Leave client_factory and async_client_factory unset. The executor sets both itself, and raises an error at startup if either is set to something else, so that a conflicting setting cannot silently send tasks to the wrong cluster.

Set [kubernetes_executor] pod_template_file to a YAML file describing the worker pod, since most setups need one. Without one, the worker pod is built from worker_container_repository and worker_container_tag alone, which only works if that image already has everything the task needs. The template’s first container must be named base and run your Airflow worker image. The pod_template_file section of the Kubernetes executor docs covers the full requirements, and its example templates work here unchanged.

Logging

Airflow tasks executed via this executor run within the configured VPC. This means that logs are not directly accessible to the Airflow UI, after task completion, the logs would be permanently lost.

Remote logging should be employed when using the EKS executor to persist your Airflow Task logs and make them viewable from the Airflow UI.

Configuring Remote Logging

There are many ways to configure remote logging and several supported destinations. A general overview of Airflow Task logging can be found here. Instructions for configuring S3 remote logging can be found here and Cloudwatch remote logging here. Some important things to point out for remote logging in the context of the EKS executor:

  • Worker pods are deleted when their task succeeds ([kubernetes_executor] delete_worker_pods), and their logs go with them, so configure remote logging to CloudWatch Logs or S3 to keep task logs viewable in the Airflow UI.

  • The remote logging configuration must be set on the worker pods as well as on the scheduler and API server, and the worker pods need an IAM role, for example through EKS Pod Identity, that can write to the log destination.

Setting up an EKS Executor for Apache Airflow

Grant access to the cluster

Create an EKS access entry for the IAM role or user behind the executor’s AWS connection ([aws_eks_executor] conn_id, aws_default by default), and associate a policy that allows create, get, list, watch, patch and delete on pods, and get on pods/log, in the namespace you plan to use. If you manage cluster access through the aws-auth ConfigMap instead, add the principal there and map it to a Kubernetes group with the same permissions.

Configure Airflow

Select the executor and name your cluster:

[core]
executor = airflow.providers.amazon.aws.executors.eks.eks_executor.AwsEksExecutor

[aws_eks_executor]
cluster_name = airflow-eks-cluster
region_name = us-east-1
conn_id = aws_default

[kubernetes_executor]
namespace = airflow
pod_template_file = /opt/airflow/pod_templates/worker_template.yaml
worker_container_repository = my-account.dkr.ecr.us-east-1.amazonaws.com/airflow
worker_container_tag = latest

The worker image must contain Airflow and the Amazon provider, and it needs access to your Dag files and a network path to the Airflow API server, in the same way any Kubernetes executor worker does. Configure remote logging as described in the logging section.

Start the scheduler and trigger a small Dag. A worker pod should appear in your namespace and the task should finish. If the scheduler logs an authentication or forbidden error from the Kubernetes API, check the in-cluster access grant for your IAM principal first.

Multi-team deployments

The executor supports multi-team mode the same way the Kubernetes executor does. Teams share the cluster set in the global [aws_eks_executor] section, and each team gets its own team-scoped [kubernetes_executor] settings, such as the namespace and pod template.

The executor refuses to start next to a Kubernetes executor in the same team, or when both are global, because that Kubernetes executor would also send its pods to the EKS cluster; configure them for different teams instead (see #74844).

Fault tolerance

Fault tolerance, including how worker pod crashes and scheduler restarts are handled, is the same as for the Kubernetes executor; see Kubernetes Executor. Because tokens are replaced before they expire, repeated authentication failures on a long-running scheduler usually mean the credentials behind your connection are no longer valid, or the role has lost its access entry on the cluster.

Was this entry helpful?