airflow.providers.cncf.kubernetes.triggers.kueue

Wait for a Kubernetes Job run through Kueue.

Attributes

JOB_STATE_MESSAGES

Classes

KueueJobState

State of a Job run through Kueue.

KubernetesKueueJobTrigger

Wait until a Job run through Kueue is finished, even if Kueue preempts it several times.

Functions

get_job_state(job)

Return the state of a Job run through Kueue; finished means Complete or Failed.

build_pod_label_selector(job)

Return the label selector of the pods of the Job, built from its spec.selector.

build_job_event(job, xcom_results)

Build the event of a finished Job, as passed to KubernetesStartKueueJobOperator.execute_complete.

get_succeeded_pod_xcoms(pods, xcom_results)

Return the XComs read from the pods that succeeded, skipping failed and preempted pods.

list_job_pods(client, namespace, label_selector, job_uid)

Return the pods the Job has now: those matched by build_pod_label_selector that the Job owns.

is_pod_deleted(client, pod)

Return whether the pod no longer exists.

collect_xcom_results(client, pod_manager, job, ...)

Read into xcom_results the XCom of each pod of the Job whose base container has ended.

Module Contents

class airflow.providers.cncf.kubernetes.triggers.kueue.KueueJobState[source]

Bases: str, enum.Enum

State of a Job run through Kueue.

SUSPENDED = 'suspended'[source]
RUNNING = 'running'[source]
FINISHED = 'finished'[source]
airflow.providers.cncf.kubernetes.triggers.kueue.JOB_STATE_MESSAGES[source]
airflow.providers.cncf.kubernetes.triggers.kueue.get_job_state(job)[source]

Return the state of a Job run through Kueue; finished means Complete or Failed.

airflow.providers.cncf.kubernetes.triggers.kueue.build_pod_label_selector(job)[source]

Return the label selector of the pods of the Job, built from its spec.selector.

Kubernetes adds the batch.kubernetes.io/job-name label to the pods only without manualSelector, while spec.selector matches them in both modes.

airflow.providers.cncf.kubernetes.triggers.kueue.build_job_event(job, xcom_results)[source]

Build the event of a finished Job, as passed to KubernetesStartKueueJobOperator.execute_complete.

Parameters:
  • job (kubernetes.client.models.V1Job) – the finished Job.

  • xcom_results (dict[str, str]) – the raw XComs read from the pods, by pod name.

airflow.providers.cncf.kubernetes.triggers.kueue.get_succeeded_pod_xcoms(pods, xcom_results)[source]

Return the XComs read from the pods that succeeded, skipping failed and preempted pods.

airflow.providers.cncf.kubernetes.triggers.kueue.list_job_pods(client, namespace, label_selector, job_uid)[source]

Return the pods the Job has now: those matched by build_pod_label_selector that the Job owns.

With manualSelector, the selector may also match the pods of other Jobs, such as the same task in another Dag run; the Job controller tells them apart by owner too.

airflow.providers.cncf.kubernetes.triggers.kueue.is_pod_deleted(client, pod)[source]

Return whether the pod no longer exists.

airflow.providers.cncf.kubernetes.triggers.kueue.collect_xcom_results(client, pod_manager, job, base_container_name, xcom_results)[source]

Read into xcom_results the XCom of each pod of the Job whose base container has ended.

Reading the XCom also stops the XCom sidecar: until then the pod, and so the Job, cannot finish. Pods already read are skipped, so this is called at each poll.

class airflow.providers.cncf.kubernetes.triggers.kueue.KubernetesKueueJobTrigger(job_name, job_namespace, base_container_name, kubernetes_conn_id=None, poll_interval=10.0, cluster_context=None, config_file=None, in_cluster=None, do_xcom_push=False)[source]

Bases: airflow.triggers.base.BaseTrigger

Wait until a Job run through Kueue is finished, even if Kueue preempts it several times.

Unlike KubernetesJobTrigger, it polls the Job rather than the pods found when the Job started, and reads the XComs from the pods the Job has at each poll.

Parameters:
  • job_name (str) – The name of the job.

  • job_namespace (str) – The namespace of the job.

  • base_container_name (str) – The name of the base container in the pod.

  • kubernetes_conn_id (str | None) – The kubernetes connection id for the Kubernetes cluster.

  • cluster_context (str | None) – Context that points to kubernetes cluster.

  • config_file (str | None) – Path to kubeconfig file.

  • poll_interval (float) – Polling period in seconds to check for the status.

  • in_cluster (bool | None) – run kubernetes client with in_cluster configuration.

  • do_xcom_push (bool) – If True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.

job_name[source]
job_namespace[source]
base_container_name[source]
kubernetes_conn_id = None[source]
poll_interval = 10.0[source]
cluster_context = None[source]
config_file = None[source]
in_cluster = None[source]
do_xcom_push = False[source]
serialize()[source]

Serialize KubernetesKueueJobTrigger arguments and classpath.

async run()[source]

Poll the Job until it is finished and yield a TriggerEvent.

property hook: airflow.providers.cncf.kubernetes.hooks.kubernetes.AsyncKubernetesHook[source]
property sync_hook: airflow.providers.cncf.kubernetes.hooks.kubernetes.KubernetesHook[source]
property pod_manager: airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager[source]
async cleanup()[source]

Release the hook’s cached API client when the trigger exits.

Was this entry helpful?