tests.system.google.cloud.dataflow.example_dataflow_yaml

Example Airflow DAG for Google Cloud Dataflow YAML service.

Requirements:

This test requires gcloud command (Google Cloud SDK) to be installed on the Airflow worker <https://cloud.google.com/sdk/docs/install>`__

Attributes

PROJECT_ID

ENV_ID

DAG_ID

REGION

DATAFLOW_YAML_JOB_NAME

BQ_DATASET

BQ_INPUT_TABLE

BQ_OUTPUT_TABLE

DATAFLOW_YAML_PIPELINE_FILE_URL

BQ_VARIABLES

BQ_VARIABLES_DEF

INSERT_ROWS_QUERY

create_bq_dataset

test_run

Module Contents

tests.system.google.cloud.dataflow.example_dataflow_yaml.PROJECT_ID[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.ENV_ID[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.DAG_ID = 'dataflow_yaml'[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.REGION = 'europe-west2'[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.DATAFLOW_YAML_JOB_NAME = ''[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.BQ_DATASET = ''[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.BQ_INPUT_TABLE = ''[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.BQ_OUTPUT_TABLE = ''[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.DATAFLOW_YAML_PIPELINE_FILE_URL = 'gs://airflow-system-tests-resources/dataflow/yaml/example_beam_yaml_bq.yaml'[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.BQ_VARIABLES[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.BQ_VARIABLES_DEF[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.INSERT_ROWS_QUERY = "INSERT . VALUES ('John Doe', 900, 'USA'), ('Alice Storm', 1200, 'Australia'),('Bob Max', 1000,...[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.create_bq_dataset[source]
tests.system.google.cloud.dataflow.example_dataflow_yaml.test_run[source]

Was this entry helpful?