Supported classes¶
Below is a list of Operators and Hooks that support OpenLineage extraction, along with specific DB types that are compatible with the SQLExecuteQueryOperator.
Important
While we strive to keep the list of supported classes current, please be aware that our updating process is automated and may not always capture everything accurately. Detecting hook level lineage is challenging so make sure to double check the information provided below.
Tip
You can easily implement OpenLineage support for any operator. See Implementing OpenLineage in Operators.
Core operators¶
At the moment, two core operators support OpenLineage. These operators function as a ‘black box,’ capable of running any code, which might limit the extent of lineage extraction (e.g. lineage will usually not contain input/output datasets). To enhance the extraction of lineage information, operators can utilize the hooks listed below that support OpenLineage.
Spark operators¶
The OpenLineage integration can automatically inject information into Spark application properties when its being submitted from Airflow. The following is a list of supported operators along with the corresponding information that can be injected. See automatic injection of parent job information for more details.
apache-airflow-providers-apache-livy¶
LivyOperatorParent Job Information
Transport Information (only HTTP transport is supported for now (with api_key auth, if any))
apache-airflow-providers-apache-spark¶
SparkSubmitOperatorParent Job Information
Transport Information (only HTTP transport is supported for now (with api_key auth, if any))
apache-airflow-providers-google¶
DataprocCreateBatchOperatorParent Job Information
Transport Information (only HTTP transport is supported for now (with api_key auth, if any))
DataprocInstantiateInlineWorkflowTemplateOperatorParent Job Information
Transport Information (only HTTP transport is supported for now (with api_key auth, if any))
DataprocSubmitJobOperatorParent Job Information
Transport Information (only HTTP transport is supported for now (with api_key auth, if any))
SQL operators¶
Operators inheriting from BaseSQLOperator may be supported
out of the box. These operators can use SQL parsing and may query DB for lineage extraction.
To extract unique data from each database type, a dedicated Hook implementing OpenLineage methods is required.
Not all subclasses of BaseSQLOperator are automatically supported,
only those also using a supported Hook and similar attribute naming convention (e.g., storing query under self.sql).
Important
The level of OpenLineage extraction may vary between SQL operators. Most will provide the executed SQL text, while others may also expose additional metadata such as query IDs or other query-related information. Due to the automatic generation of this documentation, some operators listed as supported SQL operators may not contain full lineage information. Please review the implementation of your operator and its corresponding hook to confirm the level of OpenLineage support.
Currently, the following databases (hooks) are supported together with SQL operators:
Databricks (via
DatabricksSqlHook)MsSql (via
MsSqlHook)MySql (via
MySqlHook)Oracle (via
OracleHook)PgVector (via
PgVectorHook)Postgres (via
PostgresHook)Redshift (via
RedshiftSQLHook)Snowflake (via
SnowflakeHook)SnowflakeApi (via
SnowflakeSqlApiHook)Spanner (via
SpannerHook)Trino (via
TrinoHook)
Providers¶
The operators and hooks listed below from each provider are natively equipped with OpenLineage support.