airflow.providers.google.cloud.sensors.bigquery¶

This module contains Google BigQuery sensors.

Classes¶

`BigQueryTableExistenceSensor`	Checks for the existence of a table in Google Bigquery.
`BigQueryRoutineExistenceSensor`	Checks for the existence of a routine (UDF, procedure, or TVF) in a BigQuery dataset.
`BigQueryTablePartitionExistenceSensor`	Checks for the existence of a partition within a table in Google Bigquery.
`BigQueryStreamingBufferEmptySensor`	Wait for the streaming buffer of a BigQuery table to be empty.

Module Contents¶

class airflow.providers.google.cloud.sensors.bigquery.BigQueryTableExistenceSensor(*, project_id, dataset_id, table_id, gcp_conn_id='google_cloud_default', impersonation_chain=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseSensorOperator

Checks for the existence of a table in Google Bigquery.

Parameters:

project_id (str) – The Google cloud project in which to look for the table. The connection supplied to the hook must provide access to the specified project.
dataset_id (str) – The name of the dataset in which to look for the table. storage bucket.
table_id (str) – The name of the table to check the existence of.
gcp_conn_id (str) – (Optional) The connection ID used to connect to Google Cloud.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'dataset_id', 'table_id', 'impersonation_chain', 'gcp_conn_id')[source]¶

ui_color = '#f0eee4'[source]¶

project_id[source]¶

dataset_id[source]¶

table_id[source]¶

gcp_conn_id = 'google_cloud_default'[source]¶

impersonation_chain = None[source]¶

deferrable[source]¶

poke(context)[source]¶

Override when deriving this class.

execute(context)[source]¶

Airflow runs this method on the worker and defers using the trigger.

execute_complete(context, event=None)[source]¶

Act as a callback for when the trigger fires - returns immediately.

Relies on trigger to throw an exception, otherwise it assumes execution was successful.

class airflow.providers.google.cloud.sensors.bigquery.BigQueryRoutineExistenceSensor(*, project_id, dataset_id, routine_id, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseSensorOperator

Checks for the existence of a routine (UDF, procedure, or TVF) in a BigQuery dataset.

Parameters:

project_id (str) – The Google Cloud project that owns the dataset.
dataset_id (str) – The dataset that owns the routine.
routine_id (str) – The identifier of the routine to check.
gcp_conn_id (str) – (Optional) The connection ID used to connect to Google Cloud.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'dataset_id', 'routine_id', 'impersonation_chain', 'gcp_conn_id')[source]¶

ui_color = '#f0eee4'[source]¶

project_id[source]¶

dataset_id[source]¶

routine_id[source]¶

gcp_conn_id = 'google_cloud_default'[source]¶

impersonation_chain = None[source]¶

poke(context)[source]¶

Override when deriving this class.

class airflow.providers.google.cloud.sensors.bigquery.BigQueryTablePartitionExistenceSensor(*, project_id, dataset_id, table_id, partition_id, gcp_conn_id='google_cloud_default', impersonation_chain=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseSensorOperator

Checks for the existence of a partition within a table in Google Bigquery.

Parameters:

project_id (str) – The Google cloud project in which to look for the table. The connection supplied to the hook must provide access to the specified project.
dataset_id (str) – The name of the dataset in which to look for the table. storage bucket.
table_id (str) – The name of the table to check the existence of.
partition_id (str) – The name of the partition to check the existence of.
gcp_conn_id (str) – (Optional) The connection ID used to connect to Google Cloud.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'dataset_id', 'table_id', 'partition_id', 'impersonation_chain', 'gcp_conn_id')[source]¶

ui_color = '#f0eee4'[source]¶

project_id[source]¶

dataset_id[source]¶

table_id[source]¶

partition_id[source]¶

gcp_conn_id = 'google_cloud_default'[source]¶

impersonation_chain = None[source]¶

deferrable[source]¶

poke(context)[source]¶

Override when deriving this class.

execute(context)[source]¶

Airflow runs this method on the worker and defers using the triggers if deferrable is True.

execute_complete(context, event=None)[source]¶

Act as a callback for when the trigger fires - returns immediately.

Relies on trigger to throw an exception, otherwise it assumes execution was successful.

class airflow.providers.google.cloud.sensors.bigquery.BigQueryStreamingBufferEmptySensor(*, project_id, dataset_id, table_id, gcp_conn_id='google_cloud_default', impersonation_chain=None, empty_confirmations=2, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.common.compat.sdk.BaseSensorOperator

Wait for the streaming buffer of a BigQuery table to be empty.

BigQuery DML statements (UPDATE, DELETE, MERGE) cannot run against rows that are still in the streaming buffer; the buffer is flushed within ~90 minutes. Use this sensor between a streaming insert and a DML step to avoid UPDATE/MERGE/DELETE statement over table ... would affect rows in the streaming buffer errors.

Warning

BigQuery’s table.streaming_buffer metadata is eventually consistent. The sensor mitigates this by requiring multiple consecutive empty observations before reporting the buffer empty (configurable via empty_confirmations), but this may delay success by up to one poke_interval compared to earlier releases. mode="reschedule" with empty_confirmations > 1 needs a running triggerer once an empty reading is seen; empty_confirmations=1 keeps the worker-only behaviour.

Parameters:

project_id (str) – Google Cloud project containing the table.
dataset_id (str) – Dataset of the table to monitor.
table_id (str) – Table to monitor.
gcp_conn_id (str) – Airflow connection ID for GCP.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate, or a chained list of accounts. See the Google provider docs for details.
empty_confirmations (int) – Number of consecutive empty readings required before considering the streaming buffer empty. Must be at least 1. Defaults to 2.
deferrable (bool) – Run in deferrable mode using BigQueryStreamingBufferEmptyTrigger.

template_fields: collections.abc.Sequence[str] = ('project_id', 'dataset_id', 'table_id', 'impersonation_chain', 'gcp_conn_id')[source]¶

ui_color = '#f0eee4'[source]¶

project_id[source]¶

dataset_id[source]¶

table_id[source]¶

gcp_conn_id = 'google_cloud_default'[source]¶

impersonation_chain = None[source]¶

empty_confirmations = 2[source]¶

deferrable[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶

poke(context)[source]¶

Override when deriving this class.