Customizing Workers¶
Both CeleryExecutor and KubernetesExecutor workers can be highly customized with the workers parameters.
For example, to set resources on workers for CeleryExecutor:
workers:
celery:
resources:
requests:
cpu: 1
limits:
cpu: 1
Custom pod_template_file¶
With KubernetesExecutor you can also provide a complete pod_template_file
to fully override default Kubernetes workers configuration. This may be useful if you need different configuration between
worker types for KubernetesExecutor or if you need to customize something not possible with workers parameters alone.
Note
Configuration options between Celery and Kubernetes workers can be overwritten by workers.celery and workers.kubernetes sections.
As an example, let’s say you want to set priorityClassName on your workers:
Note
The following example is NOT functional, but meant to be illustrative of how you can provide a custom pod_template_file.
You’re better off starting with the default pod_template_file instead.
podTemplate: |
apiVersion: v1
kind: Pod
metadata:
name: placeholder-name
labels:
tier: airflow
component: worker
release: {{ .Release.Name }}
spec:
priorityClassName: high-priority
containers:
- name: base
The same can be achieved with default pod_template_file by overriding the priorityClassName option for KubernetesExecutor like:
workers:
kubernetes:
priorityClassName: high-priority