airflow.providers.cncf.kubernetes.pod_generator

Pod generator.

This module provides an interface between the previous Pod API and outputs a kubernetes.client.models.V1Pod. The advantage being that the full Kubernetes API is supported and no serialization need be written.

Attributes

log

MAX_LABEL_LEN

Classes

PodGenerator

Contains Kubernetes Airflow Worker configuration logic.

Functions

make_safe_label_value(string)

Normalize a provided label to be of valid length and characters.

datetime_to_label_safe_datestring(datetime_obj)

Transform a datetime string to use as a label.

label_safe_datestring_to_datetime(string)

Transform a label back to a datetime object.

merge_objects(base_obj, client_obj)

Merge objects.

extend_object_field(base_obj, client_obj, field_name)

Add field values to existing objects.

Module Contents

airflow.providers.cncf.kubernetes.pod_generator.log[source]
airflow.providers.cncf.kubernetes.pod_generator.MAX_LABEL_LEN = 63[source]
airflow.providers.cncf.kubernetes.pod_generator.make_safe_label_value(string)[source]

Normalize a provided label to be of valid length and characters.

Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.

If the label value is greater than 63 chars once made safe, or differs in any way from the original value sent to this function, then we need to truncate to 53 chars, and append it with a unique hash.

airflow.providers.cncf.kubernetes.pod_generator.datetime_to_label_safe_datestring(datetime_obj)[source]

Transform a datetime string to use as a label.

Kubernetes doesn’t like “:” in labels, since ISO datetime format uses “:” but not “_” let’s replace “:” with “_”

Parameters:

datetime_obj (datetime.datetime) – datetime.datetime object

Returns:

ISO-like string representing the datetime

Return type:

str

airflow.providers.cncf.kubernetes.pod_generator.label_safe_datestring_to_datetime(string)[source]

Transform a label back to a datetime object.

Kubernetes doesn’t permit “:” in labels. ISO datetime format uses “:” but not “_”, let’s replace “:” with “_”

Parameters:

string (str) – str

Returns:

datetime.datetime object

Return type:

datetime.datetime

class airflow.providers.cncf.kubernetes.pod_generator.PodGenerator(pod=None, pod_template_file=None, extract_xcom=True)[source]

Contains Kubernetes Airflow Worker configuration logic.

Represents a kubernetes pod and manages execution of a single pod. Any configuration that is container specific gets applied to the first container in the list of containers.

Parameters:

  • pod (kubernetes.client.models.V1Pod | None) – The fully specified pod. Mutually exclusive with pod_template_file

  • pod_template_file (str | None) – Path to YAML file. Mutually exclusive with pod

  • extract_xcom (bool) – Whether to bring up a container for xcom

extract_xcom = True[source]
static from_obj(obj)[source]

Convert to pod from obj.

static reconcile_pods(base_pod, client_pod)[source]

Merge Kubernetes Pod objects.

Parameters:

  • base_pod (kubernetes.client.models.V1Pod) – has the base attributes which are overwritten if they exist in the client pod and remain if they do not exist in the client_pod

  • client_pod (kubernetes.client.models.V1Pod | None) – the pod that the client wants to create.

Returns:

the merged pods

Return type:

kubernetes.client.models.V1Pod

This can’t be done recursively as certain fields are overwritten and some are concatenated.

static reconcile_metadata(base_meta, client_meta)[source]

Merge Kubernetes Metadata objects.

Parameters:

  • base_meta – has the base attributes which are overwritten if they exist in the client_meta and remain if they do not exist in the client_meta

  • client_meta – the spec that the client wants to create.

Returns:

the merged specs

static reconcile_specs(base_spec, client_spec)[source]

Merge Kubernetes PodSpec objects.

Parameters:

  • base_spec (kubernetes.client.models.V1PodSpec | None) – has the base attributes which are overwritten if they exist in the client_spec and remain if they do not exist in the client_spec

  • client_spec (kubernetes.client.models.V1PodSpec | None) – the spec that the client wants to create.

Returns:

the merged specs

Return type:

kubernetes.client.models.V1PodSpec | None

static reconcile_containers(base_containers, client_containers)[source]

Merge Kubernetes Container objects.

Parameters:

  • base_containers (list[kubernetes.client.models.V1Container]) – has the base attributes which are overwritten if they exist in the client_containers and remain if they do not exist in the client_containers

  • client_containers (list[kubernetes.client.models.V1Container]) – the containers that the client wants to create.

Returns:

the merged containers

Return type:

list[kubernetes.client.models.V1Container]

The runs recursively over the list of containers.

classmethod construct_pod(dag_id, task_id, pod_id, try_number, kube_image, date, args, pod_override_object, base_worker_pod, namespace, scheduler_job_id, run_id=None, map_index=-1, *, with_mutation_hook=False)[source]

Create a Pod.

Construct a pod by gathering and consolidating the configuration from 3 places:

  • airflow.cfg

  • executor_config

  • dynamic arguments

static serialize_pod(pod)[source]

Convert a k8s.V1Pod into a json serializable dictionary.

Parameters:

pod (kubernetes.client.models.V1Pod) – k8s.V1Pod object

Returns:

Serialized version of the pod returned as dict

Return type:

dict

static deserialize_model_file(path)[source]

Generate a Pod from a file.

Parameters:

path (str) – Path to the file

Returns:

a kubernetes.client.models.V1Pod

Return type:

kubernetes.client.models.V1Pod

static deserialize_model_dict(pod_dict)[source]

Deserializes a Python dictionary to k8s.V1Pod.

Unfortunately we need access to the private method _ApiClient__deserialize_model from the kubernetes client. This issue is tracked here; https://github.com/kubernetes-client/python/issues/977.

Parameters:

pod_dict (dict | None) – Serialized dict of k8s.V1Pod object

Returns:

De-serialized k8s.V1Pod

Return type:

kubernetes.client.models.V1Pod

airflow.providers.cncf.kubernetes.pod_generator.merge_objects(base_obj, client_obj)[source]

Merge objects.

Parameters:

  • base_obj – has the base attributes which are overwritten if they exist in the client_obj and remain if they do not exist in the client_obj

  • client_obj – the object that the client wants to create.

Returns:

the merged objects

airflow.providers.cncf.kubernetes.pod_generator.extend_object_field(base_obj, client_obj, field_name)[source]

Add field values to existing objects.

Parameters:

  • base_obj – an object which has a property field_name that is a list

  • client_obj – an object which has a property field_name that is a list. A copy of this object is returned with field_name modified

  • field_name – the name of the list field

Returns:

the client_obj with the property field_name being the two properties appended

Was this entry helpful?