airflow.providers.google.cloud.hooks.managed_kafka¶

This module contains a Managed Service for Apache Kafka hook.

Classes¶

ManagedKafkaHook

Hook for Managed Service for Apache Kafka APIs.

Module Contents¶

class airflow.providers.google.cloud.hooks.managed_kafka.ManagedKafkaHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Managed Service for Apache Kafka APIs.

get_managed_kafka_client()[source]¶

Return ManagedKafkaClient object.

wait_for_operation(operation, timeout=None)[source]¶

Wait for long-lasting operation to complete.

create_cluster(project_id, location, cluster, cluster_id, request_id=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Create a new Apache Kafka cluster.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster (google.cloud.managedkafka_v1.types.Cluster | dict) – Required. Configuration of the cluster to create. Its name field is ignored.
cluster_id (str) – Required. The ID to use for the cluster, which will become the final component of the cluster’s name. The ID must be 1-63 characters long, and match the regular expression [a-z]([-a-z0-9]*[a-z0-9])? to comply with RFC 1035. This value is structured like: my-cluster-id.
request_id (str | None) – Optional. An optional request ID to identify requests. Specify a unique request ID to avoid duplication of requests. If a request times out or fails, retrying with the same ID allows the server to recognize the previous attempt. For at least 60 minutes, the server ignores duplicate requests bearing the same ID. For example, consider a situation where you make an initial request and the request times out. If you make the request again with the same request ID within 60 minutes of the last request, the server checks if an original operation with the same request ID was received. If so, the server ignores the second request. The request ID must be a valid UUID. A zero UUID is not supported (00000000-0000-0000-0000-000000000000).
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

list_clusters(project_id, location, page_size=None, page_token=None, filter=None, order_by=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

List the clusters in a given project and location.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
page_size (int | None) – Optional. The maximum number of clusters to return. The service may return fewer than this value. If unspecified, server will pick an appropriate default.
page_token (str | None) – Optional. A page token, received from a previous ListClusters call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListClusters must match the call that provided the page token.
filter (str | None) – Optional. Filter expression for the result.
order_by (str | None) – Optional. Order by fields for the result.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

get_cluster(project_id, location, cluster_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Return the properties of a single cluster.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose configuration to return.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

update_cluster(project_id, location, cluster_id, cluster, update_mask, request_id=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Update the properties of a single cluster.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose configuration to update.
cluster (google.cloud.managedkafka_v1.types.Cluster | dict) – Required. The cluster to update.
update_mask (google.protobuf.field_mask_pb2.FieldMask | dict) – Required. Field mask is used to specify the fields to be overwritten in the cluster resource by the update. The fields specified in the update_mask are relative to the resource, not the full request. A field will be overwritten if it is in the mask.
request_id (str | None) – Optional. An optional request ID to identify requests. Specify a unique request ID to avoid duplication of requests. If a request times out or fails, retrying with the same ID allows the server to recognize the previous attempt. For at least 60 minutes, the server ignores duplicate requests bearing the same ID. For example, consider a situation where you make an initial request and the request times out. If you make the request again with the same request ID within 60 minutes of the last request, the server checks if an original operation with the same request ID was received. If so, the server ignores the second request. The request ID must be a valid UUID. A zero UUID is not supported (00000000-0000-0000-0000-000000000000).
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

delete_cluster(project_id, location, cluster_id, request_id=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

Delete a single cluster.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster to delete.
request_id (str | None) – Optional. An optional request ID to identify requests. Specify a unique request ID to avoid duplication of requests. If a request times out or fails, retrying with the same ID allows the server to recognize the previous attempt. For at least 60 minutes, the server ignores duplicate requests bearing the same ID. For example, consider a situation where you make an initial request and the request times out. If you make the request again with the same request ID within 60 minutes of the last request, the server checks if an original operation with the same request ID was received. If so, the server ignores the second request. The request ID must be a valid UUID. A zero UUID is not supported (00000000-0000-0000-0000-000000000000).
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

create_topic(project_id, location, cluster_id, topic_id, topic, retry=DEFAULT, timeout=None, metadata=())[source]¶

Create a new topic in a given project and location.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster in which to create the topic.
topic_id (str) – Required. The ID to use for the topic, which will become the final component of the topic’s name.
topic (google.cloud.managedkafka_v1.types.Topic | dict) – Required. Configuration of the topic to create.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

list_topics(project_id, location, cluster_id, page_size=None, page_token=None, retry=DEFAULT, timeout=None, metadata=())[source]¶

List the topics in a given cluster.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topics are to be listed.
page_size (int | None) – Optional. The maximum number of topics to return. The service may return fewer than this value. If unset or zero, all topics for the parent is returned.
page_token (str | None) – Optional. A page token, received from a previous ListTopics call. Provide this to retrieve the subsequent page. When paginating, all other parameters provided to ListTopics must match the call that provided the page token.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

get_topic(project_id, location, cluster_id, topic_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Return the properties of a single topic.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topic is to be returned.
topic_id (str) – Required. The ID of the topic whose configuration to return.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

update_topic(project_id, location, cluster_id, topic_id, topic, update_mask, retry=DEFAULT, timeout=None, metadata=())[source]¶

Update the properties of a single topic.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topic is to be updated.
topic_id (str) – Required. The ID of the topic whose configuration to update.
topic (google.cloud.managedkafka_v1.types.Topic | dict) – Required. The topic to update. Its name field must be populated.
update_mask (google.protobuf.field_mask_pb2.FieldMask | dict) – Required. Field mask is used to specify the fields to be overwritten in the Topic resource by the update. The fields specified in the update_mask are relative to the resource, not the full request. A field will be overwritten if it is in the mask.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

delete_topic(project_id, location, cluster_id, topic_id, retry=DEFAULT, timeout=None, metadata=())[source]¶

Delete a single topic.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud region that the service belongs to.
cluster_id (str) – Required. The ID of the cluster whose topic is to be deleted.
topic_id (str) – Required. The ID of the topic to delete.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.