airflow.providers.cncf.kubernetes.triggers.kueue¶
Wait for a Kubernetes Job run through Kueue.
Attributes¶
Classes¶
State of a Job run through Kueue. |
|
Wait until a Job run through Kueue is finished, even if Kueue preempts it several times. |
Functions¶
|
Return the state of a Job run through Kueue; finished means Complete or Failed. |
Return the label selector of the pods of the Job, built from its |
|
|
Build the event of a finished Job, as passed to |
|
Return the XComs read from the pods that succeeded, skipping failed and preempted pods. |
|
Return the pods the Job has now: those matched by |
|
Return whether the pod no longer exists. |
|
Read into |
Module Contents¶
- class airflow.providers.cncf.kubernetes.triggers.kueue.KueueJobState[source]¶
-
State of a Job run through Kueue.
- airflow.providers.cncf.kubernetes.triggers.kueue.get_job_state(job)[source]¶
Return the state of a Job run through Kueue; finished means Complete or Failed.
- airflow.providers.cncf.kubernetes.triggers.kueue.build_pod_label_selector(job)[source]¶
Return the label selector of the pods of the Job, built from its
spec.selector.Kubernetes adds the
batch.kubernetes.io/job-namelabel to the pods only withoutmanualSelector, whilespec.selectormatches them in both modes.
- airflow.providers.cncf.kubernetes.triggers.kueue.build_job_event(job, xcom_results)[source]¶
Build the event of a finished Job, as passed to
KubernetesStartKueueJobOperator.execute_complete.
- airflow.providers.cncf.kubernetes.triggers.kueue.get_succeeded_pod_xcoms(pods, xcom_results)[source]¶
Return the XComs read from the pods that succeeded, skipping failed and preempted pods.
- airflow.providers.cncf.kubernetes.triggers.kueue.list_job_pods(client, namespace, label_selector, job_uid)[source]¶
Return the pods the Job has now: those matched by
build_pod_label_selectorthat the Job owns.With
manualSelector, the selector may also match the pods of other Jobs, such as the same task in another Dag run; the Job controller tells them apart by owner too.
- airflow.providers.cncf.kubernetes.triggers.kueue.is_pod_deleted(client, pod)[source]¶
Return whether the pod no longer exists.
- airflow.providers.cncf.kubernetes.triggers.kueue.collect_xcom_results(client, pod_manager, job, base_container_name, xcom_results)[source]¶
Read into
xcom_resultsthe XCom of each pod of the Job whose base container has ended.Reading the XCom also stops the XCom sidecar: until then the pod, and so the Job, cannot finish. Pods already read are skipped, so this is called at each poll.
- class airflow.providers.cncf.kubernetes.triggers.kueue.KubernetesKueueJobTrigger(job_name, job_namespace, base_container_name, kubernetes_conn_id=None, poll_interval=10.0, cluster_context=None, config_file=None, in_cluster=None, do_xcom_push=False)[source]¶
Bases:
airflow.triggers.base.BaseTriggerWait until a Job run through Kueue is finished, even if Kueue preempts it several times.
Unlike
KubernetesJobTrigger, it polls the Job rather than the pods found when the Job started, and reads the XComs from the pods the Job has at each poll.- Parameters:
job_name (str) – The name of the job.
job_namespace (str) – The namespace of the job.
base_container_name (str) – The name of the base container in the pod.
kubernetes_conn_id (str | None) – The kubernetes connection id for the Kubernetes cluster.
cluster_context (str | None) – Context that points to kubernetes cluster.
config_file (str | None) – Path to kubeconfig file.
poll_interval (float) – Polling period in seconds to check for the status.
in_cluster (bool | None) – run kubernetes client with in_cluster configuration.
do_xcom_push (bool) – If True, the content of the file /airflow/xcom/return.json in the container will also be pushed to an XCom when the container completes.
- property sync_hook: airflow.providers.cncf.kubernetes.hooks.kubernetes.KubernetesHook[source]¶
- property pod_manager: airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager[source]¶