Configuration Reference¶

This page contains the list of all available Airflow configurations for the apache-airflow-providers-cncf-kubernetes provider that can be set in the airflow.cfg file or using environment variables.

Note

The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.

Note

For more information see Setting Configuration Options.

[kubernetes_executor]¶

api_client_retry_configuration¶

Kwargs to override the default urllib3 Retry used in the kubernetes API client

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__API_CLIENT_RETRY_CONFIGURATION
Example:: { "total": 3, "backoff_factor": 0.5 }

cluster_context¶

When running with in_cluster=False change the default cluster_context or config_file options to Kubernetes client. Leave blank these to use default behaviour like kubectl has.

Type:: string
Default:: None
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT

config_file¶

Path to the kubernetes configfile to be used when in_cluster is set to False

Type:: string
Default:: None
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE

delete_option_kwargs¶

Optional keyword arguments to pass to the delete_namespaced_pod kubernetes client core_v1_api method when using the Kubernetes Executor. This should be an object and can contain any of the options listed in the v1DeleteOptions class defined here: https://github.com/kubernetes-client/python/blob/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/models/v1_delete_options.py#L19

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_OPTION_KWARGS
Example:: {"grace_period_seconds": 10}

delete_worker_pods¶

If True, all worker pods will be deleted upon termination

Type:: string
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS

delete_worker_pods_on_failure¶

If False (and delete_worker_pods is True), failed worker pods will not be deleted so users can investigate them. This only prevents removal of worker pods where the worker itself failed, not when the task it ran failed.

Type:: string
Default:: False
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__DELETE_WORKER_PODS_ON_FAILURE

enable_tcp_keepalive¶

Enables TCP keepalive mechanism. This prevents Kubernetes API requests to hang indefinitely when idle connection is time-outed on services like cloud load balancers or firewalls.

Type:: boolean
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__ENABLE_TCP_KEEPALIVE

in_cluster¶

Use the service account kubernetes gives to pods to connect to kubernetes cluster. It’s intended for clients that expect to be running inside a pod running on kubernetes. It will raise an exception if called from a process not running in a kubernetes environment.

Type:: string
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER

kube_client_request_args¶

Keyword parameters to pass while calling a kubernetes client core_v1_api methods from Kubernetes Executor provided as a single line formatted JSON dictionary string. List of supported params are similar for all core_v1_apis, hence a single config variable for all apis. See: https://raw.githubusercontent.com/kubernetes-client/python/41f11a09995efcd0142e25946adc7591431bfb2f/kubernetes/client/api/core_v1_api.py

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__KUBE_CLIENT_REQUEST_ARGS

logs_task_metadata¶

Flag to control the information added to kubernetes executor logs for better traceability

Type:: boolean
Default:: False
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__LOGS_TASK_METADATA

multi_namespace_mode¶

Allows users to launch pods in multiple namespaces. Will require creating a cluster-role for the scheduler, or use multi_namespace_mode_namespace_list configuration.

Type:: boolean
Default:: False
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE

multi_namespace_mode_namespace_list¶

If multi_namespace_mode is True while scheduler does not have a cluster-role, give the list of namespaces where the scheduler will schedule jobs Scheduler needs to have the necessary permissions in these namespaces.

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__MULTI_NAMESPACE_MODE_NAMESPACE_LIST

namespace¶

The Kubernetes namespace where airflow workers should be created. Defaults to default

Type:: string
Default:: default
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE

pod_template_file¶

Path to the YAML pod file that forms the basis for KubernetesExecutor workers.

ssl_ca_cert¶

Path to a CA certificate to be used by the Kubernetes client to verify the server’s SSL certificate.

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__SSL_CA_CERT

task_publish_max_retries¶

The Maximum number of retries for queuing the task to the kubernetes scheduler when failing due to Kube API exceeded quota errors before giving up and marking task as failed. -1 for unlimited times.

Type:: integer
Default:: 0
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TASK_PUBLISH_MAX_RETRIES

tcp_keep_cnt¶

When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe tcp_keep_cnt number of times before a connection is considered to be broken.

Type:: integer
Default:: 6
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_CNT

tcp_keep_idle¶

When the enable_tcp_keepalive option is enabled, TCP probes a connection that has been idle for tcp_keep_idle seconds.

Type:: integer
Default:: 120
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_IDLE

tcp_keep_intvl¶

When the enable_tcp_keepalive option is enabled, if Kubernetes API does not respond to a keepalive probe, TCP retransmits the probe after tcp_keep_intvl seconds.

Type:: integer
Default:: 30
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__TCP_KEEP_INTVL

verify_ssl¶

Set this to false to skip verifying SSL certificate of Kubernetes python client.

Type:: boolean
Default:: True
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__VERIFY_SSL

worker_container_repository¶

The repository of the Kubernetes Image for the Worker to Run

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_REPOSITORY

worker_container_tag¶

The tag of the Kubernetes Image for the Worker to Run

Type:: string
Default:: ''
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG

worker_pod_pending_fatal_container_state_reasons¶

Added in version 8.1.0.

If the worker pods are in a pending state due to a fatal container state reasons, then fail the task and delete the worker pod if delete_worker_pods is True and delete_worker_pods_on_failure is True.

Type:: string
Default:: CreateContainerConfigError,ErrImagePull,CreateContainerError,ImageInspectError, InvalidImageName
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_POD_PENDING_FATAL_CONTAINER_STATE_REASONS

worker_pods_creation_batch_size¶

Number of Kubernetes Worker Pod creation calls per scheduler loop. Note that the current default of “1” will only launch a single pod per-heartbeat. It is HIGHLY recommended that users increase this number to match the tolerance of their kubernetes cluster for better performance.

Type:: string
Default:: 1
Environment Variable:: AIRFLOW__KUBERNETES_EXECUTOR__WORKER_PODS_CREATION_BATCH_SIZE

[local_kubernetes_executor]¶

This section only applies if you are using the LocalKubernetesExecutor in [core] section above

kubernetes_queue¶

Define when to send a task to KubernetesExecutor when using LocalKubernetesExecutor. When the queue of a task is the value of kubernetes_queue (default kubernetes), the task is executed via KubernetesExecutor, otherwise via LocalExecutor

Type:: string
Default:: kubernetes
Environment Variable:: AIRFLOW__LOCAL_KUBERNETES_EXECUTOR__KUBERNETES_QUEUE