airflow.providers.apache.beam.triggers.beam¶

Classes¶

`BeamPipelineBaseTrigger`	Base class for Beam Pipeline Triggers.
`BeamPythonPipelineTrigger`	Trigger to perform checking the Python pipeline status until it reaches terminate state.
`BeamJavaPipelineTrigger`	Trigger to perform checking the Java pipeline status until it reaches terminate state.

Module Contents¶

class airflow.providers.apache.beam.triggers.beam.BeamPipelineBaseTrigger(**kwargs)[source]¶

Bases: airflow.triggers.base.BaseTrigger

Base class for Beam Pipeline Triggers.

static file_has_gcs_path(file_path)[source]¶

static provide_gcs_tempfile(gcs_file, gcp_conn_id)[source]¶

Async:

class airflow.providers.apache.beam.triggers.beam.BeamPythonPipelineTrigger(variables, py_file, py_options=None, py_interpreter='python3', py_requirements=None, py_system_site_packages=False, runner='DirectRunner', gcp_conn_id='google_cloud_default')[source]¶

Bases: BeamPipelineBaseTrigger

Trigger to perform checking the Python pipeline status until it reaches terminate state.

Parameters:

variables (dict) – Variables passed to the pipeline.
py_file (str) – Path to the python file to execute.
py_options (list[str] | None) – Additional options.
py_interpreter (str) – Python version of the Apache Beam pipeline. If None, this defaults to the python3. To track python versions supported by beam and related issues check: https://issues.apache.org/jira/browse/BEAM-1251
py_requirements (list[str] | None) – Additional python package(s) to install. If a value is passed to this parameter, a new virtual environment has been created with additional packages installed. You could also install the apache-beam package if it is not installed on your system, or you want to use a different version.
py_system_site_packages (bool) – Whether to include system_site_packages in your virtualenv. See virtualenv documentation for more information. This option is only relevant if the py_requirements parameter is not None.
runner (str) – Runner on which pipeline will be run. By default, “DirectRunner” is being used. Other possible options: DataflowRunner, SparkRunner, FlinkRunner, PortableRunner. See: BeamRunnerType See: https://beam.apache.org/documentation/runners/capability-matrix/
gcp_conn_id (str) – Optional. The connection ID to use connecting to Google Cloud.

variables[source]¶

py_file[source]¶

py_options = None[source]¶

py_interpreter = 'python3'[source]¶

py_requirements = None[source]¶

py_system_site_packages = False[source]¶

runner = 'DirectRunner'[source]¶

gcp_conn_id = 'google_cloud_default'[source]¶

serialize()[source]¶

Serialize BeamPythonPipelineTrigger arguments and classpath.

async run()[source]¶

Get current pipeline status and yields a TriggerEvent.

class airflow.providers.apache.beam.triggers.beam.BeamJavaPipelineTrigger(variables, jar, job_class=None, runner='DirectRunner', gcp_conn_id='google_cloud_default')[source]¶

Bases: BeamPipelineBaseTrigger

Trigger to perform checking the Java pipeline status until it reaches terminate state.

Parameters:

variables (dict) – Variables passed to the job.
jar (str) – Name of the jar for the pipeline.
job_class (str | None) – Optional. Name of the java class for the pipeline.
runner (str) – Runner on which pipeline will be run. By default, “DirectRunner” is being used. Other possible options: DataflowRunner, SparkRunner, FlinkRunner, PortableRunner. See: BeamRunnerType See: https://beam.apache.org/documentation/runners/capability-matrix/
gcp_conn_id (str) – Optional. The connection ID to use connecting to Google Cloud.

variables[source]¶

jar[source]¶

job_class = None[source]¶

runner = 'DirectRunner'[source]¶

gcp_conn_id = 'google_cloud_default'[source]¶

serialize()[source]¶

Serialize BeamJavaPipelineTrigger arguments and classpath.

async run()[source]¶

Get current Java pipeline status and yields a TriggerEvent.