airflow.providers.google.cloud.operators.managed_kafka¶

This module contains Managed Service for Apache Kafka operators.

Classes¶

`ManagedKafkaBaseOperator`	Base class for Managed Kafka operators.
`ManagedKafkaCreateClusterOperator`	Create a new Apache Kafka cluster.
`ManagedKafkaListClustersOperator`	List the clusters in a given project and location.
`ManagedKafkaGetClusterOperator`	Get an Apache Kafka cluster.
`ManagedKafkaUpdateClusterOperator`	Update the properties of a single cluster.
`ManagedKafkaDeleteClusterOperator`	Delete an Apache Kafka cluster.
`ManagedKafkaCreateTopicOperator`	Create a new topic in a given project and location.
`ManagedKafkaListTopicsOperator`	List the topics in a given cluster.
`ManagedKafkaGetTopicOperator`	Return the properties of a single topic.
`ManagedKafkaUpdateTopicOperator`	Update the properties of a single topic.
`ManagedKafkaDeleteTopicOperator`	Delete a single topic.

Module Contents¶

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaBaseOperator(project_id, location, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶

Bases: airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator

Base class for Managed Kafka operators.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use connecting to Google Cloud.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str] = ('location', 'gcp_conn_id', 'project_id', 'impersonation_chain')[source]¶

location[source]¶

project_id[source]¶

retry[source]¶

timeout = None[source]¶

metadata = ()[source]¶

gcp_conn_id = 'google_cloud_default'[source]¶

impersonation_chain = None[source]¶

property hook: airflow.providers.google.cloud.hooks.managed_kafka.ManagedKafkaHook[source]¶

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaCreateClusterOperator(cluster, cluster_id, request_id=None, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Create a new Apache Kafka cluster.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster (google.cloud.managedkafka_v1.types.Cluster | dict) – Required. Configuration of the cluster to create. Its name field is ignored.
cluster_id (str) – Required. The ID to use for the cluster, which will become the final component of the cluster’s name. The ID must be 1-63 characters long, and match the regular expression [a-z]([-a-z0-9]*[a-z0-9])? to comply with RFC 1035. This value is structured like: my-cluster-id.
request_id (str | None) – Optional. An optional request ID to identify requests. Specify a unique request ID to avoid duplication of requests. If a request times out or fails, retrying with the same ID allows the server to recognize the previous attempt. For at least 60 minutes, the server ignores duplicate requests bearing the same ID. For example, consider a situation where you make an initial request and the request times out. If you make the request again with the same request ID within 60 minutes of the last request, the server checks if an original operation with the same request ID was received. If so, the server ignores the second request. The request ID must be a valid UUID. A zero UUID is not supported (00000000-0000-0000-0000-000000000000).
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

cluster[source]¶

cluster_id[source]¶

request_id = None[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaListClustersOperator(page_size=None, page_token=None, filter=None, order_by=None, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

List the clusters in a given project and location.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
page_size (int | None) – Optional. The maximum number of clusters to return. The service may return fewer than this value. If unspecified, server will pick an appropriate default.
page_token (str | None) – Optional. A page token, received from a previous ListClusters call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListClusters must match the call that provided the page token.
filter (str | None) – Optional. Filter expression for the result.
order_by (str | None) – Optional. Order by fields for the result.
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

page_size = None[source]¶

page_token = None[source]¶

filter = None[source]¶

order_by = None[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaGetClusterOperator(cluster_id, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Get an Apache Kafka cluster.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose configuration to return.
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

cluster_id[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaUpdateClusterOperator(cluster_id, cluster, update_mask, request_id=None, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Update the properties of a single cluster.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose configuration to update.
cluster (google.cloud.managedkafka_v1.types.Cluster | dict) – Required. The cluster to update.
update_mask (google.protobuf.field_mask_pb2.FieldMask | dict) – Required. Field mask is used to specify the fields to be overwritten in the cluster resource by the update. The fields specified in the update_mask are relative to the resource, not the full request. A field will be overwritten if it is in the mask.
request_id (str | None) – Optional. An optional request ID to identify requests. Specify a unique request ID to avoid duplication of requests. If a request times out or fails, retrying with the same ID allows the server to recognize the previous attempt. For at least 60 minutes, the server ignores duplicate requests bearing the same ID. For example, consider a situation where you make an initial request and the request times out. If you make the request again with the same request ID within 60 minutes of the last request, the server checks if an original operation with the same request ID was received. If so, the server ignores the second request. The request ID must be a valid UUID. A zero UUID is not supported (00000000-0000-0000-0000-000000000000).
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

cluster_id[source]¶

cluster[source]¶

update_mask[source]¶

request_id = None[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaDeleteClusterOperator(cluster_id, request_id=None, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Delete an Apache Kafka cluster.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster to delete.
request_id (str | None) – Optional. An optional request ID to identify requests. Specify a unique request ID to avoid duplication of requests. If a request times out or fails, retrying with the same ID allows the server to recognize the previous attempt. For at least 60 minutes, the server ignores duplicate requests bearing the same ID. For example, consider a situation where you make an initial request and the request times out. If you make the request again with the same request ID within 60 minutes of the last request, the server checks if an original operation with the same request ID was received. If so, the server ignores the second request. The request ID must be a valid UUID. A zero UUID is not supported (00000000-0000-0000-0000-000000000000).
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

cluster_id[source]¶

request_id = None[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaCreateTopicOperator(cluster_id, topic_id, topic, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Create a new topic in a given project and location.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster in which to create the topic.
topic_id (str) – Required. The ID to use for the topic, which will become the final component of the topic’s name.
topic (google.cloud.managedkafka_v1.types.Topic | dict) – Required. Configuration of the topic to create.
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

cluster_id[source]¶

topic_id[source]¶

topic[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaListTopicsOperator(cluster_id, page_size=None, page_token=None, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

List the topics in a given cluster.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topics are to be listed.
page_size (int | None) – Optional. The maximum number of topics to return. The service may return fewer than this value. If unset or zero, all topics for the parent is returned.
page_token (str | None) – Optional. A page token, received from a previous ListTopics call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListTopics must match the call that provided the page token.
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

cluster_id[source]¶

page_size = None[source]¶

page_token = None[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaGetTopicOperator(cluster_id, topic_id, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Return the properties of a single topic.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topic is to be returned.
topic_id (str) – Required. The ID of the topic whose configuration to return.
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

cluster_id[source]¶

topic_id[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaUpdateTopicOperator(cluster_id, topic_id, topic, update_mask, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Update the properties of a single topic.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topic is to be updated.
topic_id (str) – Required. The ID of the topic whose configuration to update.
topic (google.cloud.managedkafka_v1.types.Topic | dict) – Required. The topic to update. Its name field must be populated.
update_mask (google.protobuf.field_mask_pb2.FieldMask | dict) – Required. Field mask is used to specify the fields to be overwritten in the Topic resource by the update. The fields specified in the update_mask are relative to the resource, not the full request. A field will be overwritten if it is in the mask.
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

cluster_id[source]¶

topic_id[source]¶

topic[source]¶

update_mask[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.google.cloud.operators.managed_kafka.ManagedKafkaDeleteTopicOperator(cluster_id, topic_id, *args, **kwargs)[source]¶

Bases: ManagedKafkaBaseOperator

Delete a single topic.

Parameters:

project_id – Required. The ID of the Google Cloud project that the service belongs to.
location – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topic is to be deleted.
topic_id (str) – Required. The ID of the topic to delete.
retry – Designation of what errors, if any, should be retried.
timeout – The timeout for this request.
metadata – Strings which should be sent along with the request as metadata.
gcp_conn_id – The connection ID to use connecting to Google Cloud.
impersonation_chain – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).

template_fields: collections.abc.Sequence[str][source]¶

cluster_id[source]¶

topic_id[source]¶

execute(context)[source]¶

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.