tests.system.google.cloud.dataprep.example_dataprep

Example Airflow DAG that shows how to use Google Dataprep.

This DAG relies on the following OS environment variables

Attributes

ENV_ID

DAG_ID

CONNECTION_ID

DATAPREP_TOKEN

GCP_PROJECT_ID

GCS_BUCKET_NAME

GCS_BUCKET_PATH

DATASET_URI

DATASET_NAME

DATASET_WRANGLED_NAME

DATASET_WRANGLED_ID

FLOW_ID

FLOW_COPY_ID

RECIPE_NAME

WRITE_SETTINGS

log

create_bucket_task

test_run

Module Contents

tests.system.google.cloud.dataprep.example_dataprep.ENV_ID[source]
tests.system.google.cloud.dataprep.example_dataprep.DAG_ID = 'dataprep'[source]
tests.system.google.cloud.dataprep.example_dataprep.CONNECTION_ID = ''[source]
tests.system.google.cloud.dataprep.example_dataprep.DATAPREP_TOKEN[source]
tests.system.google.cloud.dataprep.example_dataprep.GCP_PROJECT_ID[source]
tests.system.google.cloud.dataprep.example_dataprep.GCS_BUCKET_NAME = 'dataprep-bucket-dataprep-Uninferable'[source]
tests.system.google.cloud.dataprep.example_dataprep.GCS_BUCKET_PATH = 'gs://dataprep-bucket-dataprep-Uninferable/task_results/'[source]
tests.system.google.cloud.dataprep.example_dataprep.DATASET_URI = 'gs://airflow-system-tests-resources/dataprep/dataset-00000.parquet'[source]
tests.system.google.cloud.dataprep.example_dataprep.DATASET_NAME = ''[source]
tests.system.google.cloud.dataprep.example_dataprep.DATASET_WRANGLED_NAME = 'wrangled_'[source]
tests.system.google.cloud.dataprep.example_dataprep.DATASET_WRANGLED_ID = "{{ task_instance.xcom_pull('create_wrangled_dataset')['id'] }}"[source]
tests.system.google.cloud.dataprep.example_dataprep.FLOW_ID = "{{ task_instance.xcom_pull('create_flow')['id'] }}"[source]
tests.system.google.cloud.dataprep.example_dataprep.FLOW_COPY_ID = "{{ task_instance.xcom_pull('copy_flow')['id'] }}"[source]
tests.system.google.cloud.dataprep.example_dataprep.RECIPE_NAME = 'wrangled_'[source]
tests.system.google.cloud.dataprep.example_dataprep.WRITE_SETTINGS[source]
tests.system.google.cloud.dataprep.example_dataprep.log[source]
tests.system.google.cloud.dataprep.example_dataprep.create_bucket_task[source]
tests.system.google.cloud.dataprep.example_dataprep.test_run[source]

Was this entry helpful?