tests.system.google.cloud.dataflow.example_dataflow_pipeline¶

Example Airflow DAG for testing Google Dataflow to create pipelines.

Attributes¶

`DAG_ID`
`ENV_ID`
`GCP_PROJECT_ID`
`GCP_LOCATION`
`PIPELINE_NAME`
`PIPELINE_JOB_NAME`
`PIPELINE_TYPE`
`RESOURCE_DATA_BUCKET`
`BUCKET_NAME`
`FILE_NAME`
`TEMPLATE_FILE`
`TEMP_LOCATION`
`GCS_PATH`
`INPUT_FILE`
`OUTPUT`
`create_bucket`
`test_run`

Module Contents¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.DAG_ID = 'dataflow_pipeline'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.ENV_ID[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.GCP_PROJECT_ID[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.GCP_LOCATION = 'us-central1'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.PIPELINE_NAME = ''[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.PIPELINE_JOB_NAME = ''[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.PIPELINE_TYPE = 'PIPELINE_TYPE_BATCH'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.RESOURCE_DATA_BUCKET = 'airflow-system-tests-resources'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.BUCKET_NAME = ''[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.FILE_NAME = 'kinglear.txt'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.TEMPLATE_FILE = 'word-count.json'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.TEMP_LOCATION = 'gs:///temp'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.GCS_PATH = 'gs:///dataflow/word-count.json'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.INPUT_FILE = 'gs:///dataflow/kinglear.txt'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.OUTPUT = 'gs:///results/hello'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.create_bucket[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline.test_run[source]¶

Was this entry helpful?