`airflow.providers.google.cloud.operators.cloud_sql`¶

This module contains Google Cloud SQL operators.

Module Contents¶

Classes¶

`CloudSQLBaseOperator`	Abstract base operator for Google Cloud SQL operators.
`CloudSQLCreateInstanceOperator`	Create a new Cloud SQL instance.
`CloudSQLInstancePatchOperator`	Update settings of a Cloud SQL instance.
`CloudSQLDeleteInstanceOperator`	Delete a Cloud SQL instance.
`CloudSQLCloneInstanceOperator`	Clone an instance to a target instance.
`CloudSQLCreateInstanceDatabaseOperator`	Create a new database inside a Cloud SQL instance.
`CloudSQLPatchInstanceDatabaseOperator`	Update resource containing information about a database using patch semantics.
`CloudSQLDeleteInstanceDatabaseOperator`	Delete a database from a Cloud SQL instance.
`CloudSQLExportInstanceOperator`	Export data from a Cloud SQL instance to a Cloud Storage bucket.
`CloudSQLImportInstanceOperator`	Import data into a Cloud SQL instance from Cloud Storage.
`CloudSQLExecuteQueryOperator`	Perform DML or DDL query on an existing Cloud Sql instance.

Attributes¶

`SETTINGS`
`SETTINGS_VERSION`
`CLOUD_SQL_CREATE_VALIDATION`
`CLOUD_SQL_EXPORT_VALIDATION`
`CLOUD_SQL_IMPORT_VALIDATION`
`CLOUD_SQL_DATABASE_CREATE_VALIDATION`
`CLOUD_SQL_DATABASE_PATCH_VALIDATION`

airflow.providers.google.cloud.operators.cloud_sql.SETTINGS = 'settings'[source]¶

airflow.providers.google.cloud.operators.cloud_sql.SETTINGS_VERSION = 'settingsVersion'[source]¶

airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_CREATE_VALIDATION: collections.abc.Sequence[dict][source]¶

airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_EXPORT_VALIDATION[source]¶

airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_IMPORT_VALIDATION[source]¶

airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_DATABASE_CREATE_VALIDATION[source]¶

airflow.providers.google.cloud.operators.cloud_sql.CLOUD_SQL_DATABASE_PATCH_VALIDATION[source]¶

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLBaseOperator(*, instance, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', impersonation_chain=None, **kwargs)[source]¶

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Abstract base operator for Google Cloud SQL operators.

Parameters

instance (str) – Cloud SQL instance ID. This does not include the project ID.
project_id (str) – Optional, Google Cloud Project ID. f set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLCreateInstanceOperator(*, body, instance, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Create a new Cloud SQL instance.

If an instance with the same name exists, no action will be taken and the operator will succeed.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLCreateInstanceOperator

Parameters

body (dict) – Body required by the Cloud SQL insert API, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/insert #request-body
instance (str) – Cloud SQL instance ID. This does not include the project ID.
project_id (str) – Optional, Google Cloud Project ID. If set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
validate_body (bool) – True if body should be validated, False otherwise.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#FADBDA'[source]¶

operator_extra_links = ()[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLInstancePatchOperator(*, body, instance, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Update settings of a Cloud SQL instance.

Caution: This is a partial update, so only included values for the settings will be updated.

In the request body, supply the relevant portions of an instance resource, according to the rules of patch semantics. https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch

See also

For more information on how to use this operator, take a look at the guide: CloudSQLInstancePatchOperator

Parameters

body (dict) – Body required by the Cloud SQL patch API, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/patch#request-body
instance (str) – Cloud SQL instance ID. This does not include the project ID.
project_id (str) – Optional, Google Cloud Project ID. If set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#FBDAC8'[source]¶

operator_extra_links = ()[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLDeleteInstanceOperator(*, instance, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Delete a Cloud SQL instance.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLDeleteInstanceOperator

Parameters

instance (str) – Cloud SQL instance ID. This does not include the project ID.
project_id (str) – Optional, Google Cloud Project ID. If set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#FEECD2'[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLCloneInstanceOperator(*, instance, destination_instance_name, clone_context=None, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Clone an instance to a target instance.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLCloneInstanceOperator

Parameters

instance (str) – Database instance ID to be cloned. This does not include the project ID.
destination_instance_name (str) – Database instance ID to be created. This does not include the project ID.
clone_context (dict | None) – additional clone_context parameters as described in https://cloud.google.com/sql/docs/mysql/admin-api/rest/v1/instances/clone
project_id (str) – Project ID of the project that contains the instance. If set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'destination_instance_name', 'gcp_conn_id', 'api_version')[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLCreateInstanceDatabaseOperator(*, instance, body, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Create a new database inside a Cloud SQL instance.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLCreateInstanceDatabaseOperator

Parameters

instance (str) – Database instance ID. This does not include the project ID.
body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/insert#request-body
project_id (str) – Optional, Google Cloud Project ID. If set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
validate_body (bool) – Whether the body should be validated. Defaults to True.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#FFFCDB'[source]¶

operator_extra_links = ()[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLPatchInstanceDatabaseOperator(*, instance, database, body, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Update resource containing information about a database using patch semantics.

See: https://cloud.google.com/sql/docs/mysql/admin-api/how-tos/performance#patch

See also

For more information on how to use this operator, take a look at the guide: CloudSQLPatchInstanceDatabaseOperator

Parameters

instance (str) – Database instance ID. This does not include the project ID.
database (str) – Name of the database to be updated in the instance.
body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/databases/patch#request-body
project_id (str) – Optional, Google Cloud Project ID.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
validate_body (bool) – Whether the body should be validated. Defaults to True.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'body', 'database', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#ECF4D9'[source]¶

operator_extra_links = ()[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLDeleteInstanceDatabaseOperator(*, instance, database, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Delete a database from a Cloud SQL instance.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLDeleteInstanceDatabaseOperator

Parameters

instance (str) – Database instance ID. This does not include the project ID.
database (str) – Name of the database to be deleted in the instance.
project_id (str) – Optional, Google Cloud Project ID. If set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'database', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#D5EAD8'[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLExportInstanceOperator(*, instance, body, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, impersonation_chain=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), poke_interval=10, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Export data from a Cloud SQL instance to a Cloud Storage bucket.

The exported format can be a SQL dump or CSV file.

Note: This operator is idempotent. If executed multiple times with the same export file URI, the export file in GCS will simply be overridden.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLExportInstanceOperator

Parameters

instance (str) – Cloud SQL instance ID. This does not include the project ID.
body (dict) – The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/export#request-body
project_id (str) – Optional, Google Cloud Project ID. If set to None or missing, the default project_id from the Google Cloud connection is used.
gcp_conn_id (str) – The connection ID used to connect to Google Cloud.
api_version (str) – API version used (e.g. v1beta4).
validate_body (bool) – Whether the body should be validated. Defaults to True.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
deferrable (bool) – Run operator in the deferrable mode.
poke_interval (int) – (Deferrable mode only) Time (seconds) to wait between calls to check the run status.

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#D4ECEA'[source]¶

operator_extra_links = ()[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶

Act as a callback for when the trigger fires - returns immediately.

Relies on trigger to throw an exception, otherwise it assumes execution was successful.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLImportInstanceOperator(*, instance, body, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', api_version='v1beta4', validate_body=True, impersonation_chain=None, **kwargs)[source]¶

Bases: CloudSQLBaseOperator

Import data into a Cloud SQL instance from Cloud Storage.

CSV IMPORT¶

This operator is NOT idempotent for a CSV import. If the same file is imported multiple times, the imported data will be duplicated in the database. Moreover, if there are any unique constraints the duplicate import may result in an error.

SQL IMPORT¶

This operator is idempotent for a SQL import if it was also exported by Cloud SQL. The exported SQL contains ‘DROP TABLE IF EXISTS’ statements for all tables to be imported.

If the import file was generated in a different way, idempotence is not guaranteed. It has to be ensured on the SQL file level.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLImportInstanceOperator

param instance: Cloud SQL instance ID. This does not include the project ID.
param body: The request body, as described in https://cloud.google.com/sql/docs/mysql/admin-api/v1beta4/instances/import#request-body
param project_id: Optional, Google Cloud Project ID. If set to None or missing, the default project_id from the Google Cloud connection is used.
param gcp_conn_id: The connection ID used to connect to Google Cloud.
param api_version: API version used (e.g. v1beta4).
param validate_body: Whether the body should be validated. Defaults to True.
param impersonation_chain: Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('project_id', 'instance', 'body', 'gcp_conn_id', 'api_version', 'impersonation_chain')[source]¶

ui_color = '#D3EDFB'[source]¶

operator_extra_links = ()[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.cloud_sql.CloudSQLExecuteQueryOperator(*, sql, autocommit=False, parameters=None, gcp_conn_id='google_cloud_default', gcp_cloudsql_conn_id='google_cloud_sql_default', sql_proxy_binary_path=None, ssl_server_cert=None, ssl_client_cert=None, ssl_client_key=None, ssl_secret_id=None, **kwargs)[source]¶

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Perform DML or DDL query on an existing Cloud Sql instance.

It optionally uses cloud-sql-proxy to establish secure connection with the database.

See also

For more information on how to use this operator, take a look at the guide: CloudSQLExecuteQueryOperator

Parameters

sql (str | collections.abc.Iterable[str]) – SQL query or list of queries to run (should be DML or DDL query - this operator does not return any data from the database, so it is useless to pass it DQL queries. Note that it is responsibility of the author of the queries to make sure that the queries are idempotent. For example you can use CREATE TABLE IF NOT EXISTS to create a table.
parameters (collections.abc.Iterable | collections.abc.Mapping[str, Any] | None) – (optional) the parameters to render the SQL query with.
autocommit (bool) – if True, each command is automatically committed. (default value: False)
gcp_conn_id (str) – The connection ID used to connect to Google Cloud for cloud-sql-proxy authentication.
gcp_cloudsql_conn_id (str) – The connection ID used to connect to Google Cloud SQL its schema should be gcpcloudsql://. See CloudSQLDatabaseHook for details on how to define gcpcloudsql:// connection.
sql_proxy_binary_path (str | None) – (optional) Path to the cloud-sql-proxy binary. is not specified or the binary is not present, it is automatically downloaded.
ssl_cert – (optional) Path to client certificate to authenticate when SSL is used. Overrides the connection field sslcert.
ssl_key – (optional) Path to client private key to authenticate when SSL is used. Overrides the connection field sslkey.
ssl_root_cert – (optional) Path to server’s certificate to authenticate when SSL is used. Overrides the connection field sslrootcert.
ssl_secret_id (str | None) –
(optional) ID of the secret in Google Cloud Secret Manager that stores SSL certificate in the format below:

{‘sslcert’: ‘’,
’sslkey’: ‘’, ‘sslrootcert’: ‘’}

Overrides the connection fields sslcert, sslkey, sslrootcert. Note that according to the Secret Manager requirements, the mentioned dict should be saved as a string, and encoded with base64. Note that this parameter is incompatible with parameters ssl_cert, ssl_key, ssl_root_cert.

template_fields: collections.abc.Sequence[str] = ('sql', 'gcp_cloudsql_conn_id', 'gcp_conn_id', 'ssl_server_cert', 'ssl_client_cert',...[source]¶

template_ext: collections.abc.Sequence[str] = ('.sql',)[source]¶

template_fields_renderers[source]¶

ui_color = '#D3DEF1'[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

hook()[source]¶

airflow.providers.google.cloud.operators.cloud_sql¶

Module Contents¶

Classes¶

Attributes¶

CSV IMPORT¶

SQL IMPORT¶

`airflow.providers.google.cloud.operators.cloud_sql`¶