tests.system.cncf.kubernetes.example_spark_kubernetes

This is an example DAG which uses SparkKubernetesOperator and SparkKubernetesSensor. In this example, we create two tasks which execute sequentially. The first task is to submit sparkApplication on Kubernetes cluster(the example uses spark-pi application). and the second task is to check the final state of the sparkApplication that submitted in the first state.

Spark-on-k8s operator is required to be already installed on Kubernetes https://github.com/GoogleCloudPlatform/spark-on-k8s-operator

Attributes

ENV_ID

DAG_ID

pi_example_path

test_run

Module Contents

tests.system.cncf.kubernetes.example_spark_kubernetes.ENV_ID[source]
tests.system.cncf.kubernetes.example_spark_kubernetes.DAG_ID = 'spark_pi'[source]
tests.system.cncf.kubernetes.example_spark_kubernetes.pi_example_path[source]
tests.system.cncf.kubernetes.example_spark_kubernetes.test_run[source]

Was this entry helpful?