Customizing Workers

Both CeleryExecutor and KubernetesExecutor workers can be highly customized with the workers parameters. For example, to set resources on workers for CeleryExecutor:

values.yaml
workers:
  celery:
    resources:
      requests:
        cpu: 1
      limits:
        cpu: 1

Separate Celery worker sets

Use workers.celery.sets to configure workers for different queues. Each set inherits workers.celery settings and can override them, including extraVolumes and extraVolumeMounts. For example, mount different existing PersistentVolumeClaims for two worker sets:

values.yaml
executor: CeleryExecutor
workers:
  celery:
    sets:
      - name: analytics
        queue: analytics
        extraVolumes:
          - name: input-data
            persistentVolumeClaim:
              claimName: analytics-input
        extraVolumeMounts:
          - name: input-data
            mountPath: /opt/airflow/input
            readOnly: true
      - name: research
        queue: research
        extraVolumes:
          - name: input-data
            persistentVolumeClaim:
              claimName: research-input
        extraVolumeMounts:
          - name: input-data
            mountPath: /opt/airflow/input
            readOnly: true

Create both PersistentVolumeClaims before deploying the chart, and route tasks to the corresponding queue. The default worker set remains enabled to consume tasks sent to the default queue. Each set inherits replica and autoscaling settings. If its pods can run on multiple nodes, use a claim that supports read-only mounts from multiple nodes, such as ReadOnlyMany, with a compatible storage driver. Non-empty list overrides such as extraVolumes and extraVolumeMounts replace the inherited list, so include all additional volumes and mounts needed by each set. An empty list retains the inherited value.

Custom pod_template_file

With KubernetesExecutor you can also provide a complete pod_template_file to fully override default Kubernetes workers configuration. This may be useful if you need different configuration between worker types for KubernetesExecutor or if you need to customize something not possible with workers parameters alone.

Note

Configuration options between Celery and Kubernetes workers can be overwritten by workers.celery and workers.kubernetes sections.

As an example, let’s say you want to set priorityClassName on your workers:

Note

The following example is NOT functional, but meant to be illustrative of how you can provide a custom pod_template_file. You’re better off starting with the default pod_template_file instead.

values.yaml
podTemplate: |
  apiVersion: v1
  kind: Pod
  metadata:
    name: placeholder-name
    labels:
      tier: airflow
      component: worker
      release: {{ .Release.Name }}
  spec:
    priorityClassName: high-priority
    containers:
      - name: base

The same can be achieved with default pod_template_file by overriding the priorityClassName option for KubernetesExecutor like:

values.yaml
workers:
  kubernetes:
    priorityClassName: high-priority

Was this entry helpful?