Google Cloud Data Catalog Operators¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. The Data Catalog operators replacement can be found at airflow.providers.google.cloud.operators.dataplex For further understanding please refer to the official guide. Mapping between entities from Data Catalog and Dataplex Universal Catalog presented in table Mapping between Data Catalog and Dataplex Universal Catalog under Learn more about simultaneous availability of Data Catalog metadata in Dataplex Universal Catalog block.
The Data Catalog is a fully managed and scalable metadata management service that allows organizations to quickly discover, manage and understand all their data in Google Cloud. It offers:
A simple and easy to use search interface for data discovery, powered by the same Google search technology that supports Gmail and Drive
A flexible and powerful cataloging system for capturing technical and business metadata
An auto-tagging mechanism for sensitive data with DLP API integration
Prerequisite Tasks¶
To use these operators, you must do a few things:
Select or create a Cloud Platform project using the Cloud Console.
Enable billing for your project, as described in the Google Cloud documentation.
Enable the API, as described in the Cloud Console documentation.
Install API libraries via pip.
pip install 'apache-airflow[google]'Detailed information is available for Installation.
Managing an entries¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogGetEntryOperator or
DataplexCatalogLookupEntryOperator.
For more information please check this section.
Operators uses a Entry for representing entry
Getting an entry¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogGetEntryOperator or
DataplexCatalogLookupEntryOperator.
For more information please check this section.
Getting an entry is performed with the
CloudDataCatalogGetEntryOperator and
CloudDataCatalogLookupEntryOperator
operators.
The CloudDataCatalogGetEntryOperator use Project ID, Entry Group ID, Entry ID to get the entry.
You can use Jinja templating with
location, entry_group, entry, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
The CloudDataCatalogLookupEntryOperator use the resource name to get the entry.
You can use Jinja templating with
linked_resource, sql_resource, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
Creating an entry¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogCreateEntryOperator.
For more information please check this section.
The CloudDataCatalogCreateEntryOperator
operator create the entry.
You can use Jinja templating with
location, entry_group, entry_id, entry, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
The newly created entry ID can be read with the entry_id key.
Updating an entry¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogUpdateEntryOperator.
For more information please check this section.
The CloudDataCatalogUpdateEntryOperator
operator update the entry.
You can use Jinja templating with
entry, update_mask, location, entry_group, entry_id, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Deleting a entry¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogDeleteEntryOperator.
For more information please check this section.
The CloudDataCatalogDeleteEntryOperator
operator delete the entry.
You can use Jinja templating with
location, entry_group, entry, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Managing a entry groups¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogCreateEntryGroupOperator.
For more information please check this section.
Operators uses a Entry for representing a entry groups.
Creating an entry group¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogCreateEntryGroupOperator.
For more information please check this section.
The CloudDataCatalogCreateEntryGroupOperator
operator create the entry group.
You can use Jinja templating with
location, entry_group_id, entry_group, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
The newly created entry group ID can be read with the entry_group_id key.
Getting an entry group¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogGetEntryGroupOperator.
For more information please check this section.
The CloudDataCatalogGetEntryGroupOperator
operator get the entry group.
You can use Jinja templating with
location, entry_group, read_mask, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
Deleting an entry group¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogDeleteEntryGroupOperator.
For more information please check this section.
The CloudDataCatalogDeleteEntryGroupOperator
operator delete the entry group.
You can use Jinja templating with
location, entry_group, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Managing tag templates¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogCreateAspectTypeOperator.
For more information please check this section.
Operators uses a TagTemplate for representing a tag templates.
Creating a tag template¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogCreateAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogCreateTagTemplateOperator
operator get the tag template.
You can use Jinja templating with
location, tag_template_id, tag_template, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
The newly created tag template ID can be read with the tag_template_id key.
Deleting a tag template¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogDeleteAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogDeleteTagTemplateOperator
operator delete the tag template.
You can use Jinja templating with
location, tag_template, force, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Getting a tag template¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogGetAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogGetTagTemplateOperator
operator get the tag template.
You can use Jinja templating with
location, tag_template, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
Updating a tag template¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogUpdateAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogUpdateTagTemplateOperator
operator update the tag template.
You can use Jinja templating with
tag_template, update_mask, location, tag_template_id, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Managing a tag template fields¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogUpdateAspectTypeOperator or
DataplexCatalogCreateAspectTypeOperator.
For more information please check this section.
Operators uses a TagTemplateField for representing a tag template fields.
Creating a field¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogUpdateAspectTypeOperator or
DataplexCatalogCreateAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogCreateTagTemplateFieldOperator
operator get the tag template field.
You can use Jinja templating with
location, tag_template, tag_template_field_id, tag_template_field, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
The newly created field ID can be read with the tag_template_field_id key.
Renaming a field¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogUpdateAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogRenameTagTemplateFieldOperator
operator rename the tag template field.
You can use Jinja templating with
location, tag_template, field, new_tag_template_field_id, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Updating a field¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogUpdateAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogUpdateTagTemplateFieldOperator
operator get the tag template field.
You can use Jinja templating with
tag_template_field, update_mask, tag_template_field_name, location, tag_template, tag_template_field_id, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Deleting a field¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogUpdateAspectTypeOperator.
For more information please check this section.
The CloudDataCatalogDeleteTagTemplateFieldOperator
operator delete the tag template field.
You can use Jinja templating with
location, tag_template, field, force, project_id, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
Search resources¶
Warning
The Data Catalog will be discontinued on January 30, 2026 in favor of Dataplex Universal Catalog. Please use
DataplexCatalogSearchEntriesOperator.
For more information please check this section.
The CloudDataCatalogSearchCatalogOperator
operator searches Data Catalog for multiple resources like entries, tags that match a query.
The query parameters should defined using search syntax.
You can use Jinja templating with
scope, query, page_size, order_by, retry, timeout, metadata, gcp_conn_id, impersonation_chain
parameters which allows you to dynamically determine values.
The result is saved to XCom, which allows it to be used by other operators.
Reference¶
For further information, look at: