airflow.providers.google.cloud.operators.datastore

This module contains Google Datastore operators.

Classes

CloudDatastoreExportEntitiesOperator

Export entities from Google Cloud Datastore to Cloud Storage.

CloudDatastoreImportEntitiesOperator

Import entities from Cloud Storage to Google Cloud Datastore.

CloudDatastoreAllocateIdsOperator

Allocate IDs for incomplete keys. Return list of keys.

CloudDatastoreBeginTransactionOperator

Begins a new transaction. Returns a transaction handle.

CloudDatastoreCommitOperator

Commit a transaction, optionally creating, deleting or modifying some entities.

CloudDatastoreRollbackOperator

Roll back a transaction.

CloudDatastoreRunQueryOperator

Run a query for entities. Returns the batch of query results.

CloudDatastoreGetOperationOperator

Gets the latest state of a long-running operation.

CloudDatastoreDeleteOperationOperator

Deletes the long-running operation.

Module Contents

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreExportEntitiesOperator(*, bucket, namespace=None, datastore_conn_id='google_cloud_default', cloud_storage_conn_id='google_cloud_default', entity_filter=None, labels=None, polling_interval_in_seconds=10, overwrite_existing=False, project_id=PROVIDE_PROJECT_ID, impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Export entities from Google Cloud Datastore to Cloud Storage.

See also

For more information on how to use this operator, take a look at the guide: Export Entities

See also

https://cloud.google.com/datastore/docs/export-import-entities

Parameters:

  • bucket (str) – name of the cloud storage bucket to back up data

  • namespace (str | None) – optional namespace path in the specified Cloud Storage bucket to back up data. If this namespace does not exist in GCS, it will be created.

  • datastore_conn_id (str) – the name of the Datastore connection id to use

  • cloud_storage_conn_id (str) – the name of the cloud storage connection id to force-write backup

  • entity_filter (dict | None) – description of what data from the project is included in the export, refer to https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter

  • labels (dict | None) – client-assigned labels for cloud storage

  • polling_interval_in_seconds (int) – number of seconds to wait before polling for execution status again

  • overwrite_existing (bool) – if the storage bucket + namespace is not empty, it will be emptied prior to exports. This enables overwriting existing backups.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('bucket', 'namespace', 'entity_filter', 'labels', 'impersonation_chain')[source]
datastore_conn_id = 'google_cloud_default'[source]
cloud_storage_conn_id = 'google_cloud_default'[source]
bucket[source]
namespace = None[source]
entity_filter = None[source]
labels = None[source]
polling_interval_in_seconds = 10[source]
overwrite_existing = False[source]
project_id = None[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreImportEntitiesOperator(*, bucket, file, namespace=None, entity_filter=None, labels=None, datastore_conn_id='google_cloud_default', polling_interval_in_seconds=10, project_id=PROVIDE_PROJECT_ID, impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Import entities from Cloud Storage to Google Cloud Datastore.

See also

For more information on how to use this operator, take a look at the guide: Import Entities

See also

https://cloud.google.com/datastore/docs/export-import-entities

Parameters:

  • bucket (str) – container in Cloud Storage to store data

  • file (str) – path of the backup metadata file in the specified Cloud Storage bucket. It should have the extension .overall_export_metadata

  • namespace (str | None) – optional namespace of the backup metadata file in the specified Cloud Storage bucket.

  • entity_filter (dict | None) – description of what data from the project is included in the export, refer to https://cloud.google.com/datastore/docs/reference/rest/Shared.Types/EntityFilter

  • labels (dict | None) – client-assigned labels for cloud storage

  • datastore_conn_id (str) – the name of the connection id to use

  • polling_interval_in_seconds (float) – number of seconds to wait before polling for execution status again

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('bucket', 'file', 'namespace', 'entity_filter', 'labels', 'impersonation_chain')[source]
datastore_conn_id = 'google_cloud_default'[source]
bucket[source]
file[source]
namespace = None[source]
entity_filter = None[source]
labels = None[source]
polling_interval_in_seconds = 10[source]
project_id = None[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreAllocateIdsOperator(*, partial_keys, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Allocate IDs for incomplete keys. Return list of keys.

See also

For more information on how to use this operator, take a look at the guide: Allocate Ids

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/allocateIds

Parameters:

  • partial_keys (list) – a list of partial keys.

  • project_id (str) – Google Cloud project ID against which to make the request.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('partial_keys', 'impersonation_chain')[source]
partial_keys[source]
gcp_conn_id = 'google_cloud_default'[source]
project_id = None[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreBeginTransactionOperator(*, transaction_options, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Begins a new transaction. Returns a transaction handle.

See also

For more information on how to use this operator, take a look at the guide: Begin transaction

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/beginTransaction

Parameters:

  • transaction_options (dict[str, Any]) – Options for a new transaction.

  • project_id (str) – Google Cloud project ID against which to make the request.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('transaction_options', 'impersonation_chain')[source]
transaction_options[source]
gcp_conn_id = 'google_cloud_default'[source]
project_id = None[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreCommitOperator(*, body, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Commit a transaction, optionally creating, deleting or modifying some entities.

See also

For more information on how to use this operator, take a look at the guide: Commit transaction

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/commit

Parameters:

  • body (dict[str, Any]) – the body of the commit request.

  • project_id (str) – Google Cloud project ID against which to make the request.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('body', 'impersonation_chain')[source]
body[source]
gcp_conn_id = 'google_cloud_default'[source]
project_id = None[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreRollbackOperator(*, transaction, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Roll back a transaction.

See also

For more information on how to use this operator, take a look at the guide: Roll back transaction

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/rollback

Parameters:

  • transaction (str) – the transaction to roll back.

  • project_id (str) – Google Cloud project ID against which to make the request.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('transaction', 'impersonation_chain')[source]
transaction[source]
gcp_conn_id = 'google_cloud_default'[source]
project_id = None[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreRunQueryOperator(*, body, project_id=PROVIDE_PROJECT_ID, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Run a query for entities. Returns the batch of query results.

See also

For more information on how to use this operator, take a look at the guide: Run query

See also

https://cloud.google.com/datastore/docs/reference/rest/v1/projects/runQuery

Parameters:

  • body (dict[str, Any]) – the body of the query request.

  • project_id (str) – Google Cloud project ID against which to make the request.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('body', 'impersonation_chain')[source]
body[source]
gcp_conn_id = 'google_cloud_default'[source]
project_id = None[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreGetOperationOperator(*, name, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Gets the latest state of a long-running operation.

See also

For more information on how to use this operator, take a look at the guide: Get operation state

See also

https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/get

Parameters:

  • name (str) – the name of the operation resource.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('name', 'impersonation_chain')[source]
name[source]
gcp_conn_id = 'google_cloud_default'[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.datastore.CloudDatastoreDeleteOperationOperator(*, name, gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Deletes the long-running operation.

See also

For more information on how to use this operator, take a look at the guide: Delete operation

See also

https://cloud.google.com/datastore/docs/reference/data/rest/v1/projects.operations/delete

Parameters:

  • name (str) – the name of the operation resource.

  • gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.

  • impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('name', 'impersonation_chain')[source]
name[source]
gcp_conn_id = 'google_cloud_default'[source]
impersonation_chain = None[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?