apache-airflow-providers-common-sql

Changelog¶

1.28.1¶

Release Date: 2025-09-22

Bug Fixes¶

fix: rows_processor in SQLInsertRowsOperator (#55290)

1.28.0¶

Release Date: 2025-09-09

Features¶

Add SQLInsertRowsOperator in common.sql provider (#55031)
Added paginated_sql_statement_clause parameter to GenericTransfer (#48866)
Add rudimentary support for psycopg3 (#52976)

Bug Fixes¶

Fixed resolving of dialect name when host of JdbcHook is an JDBC URL (#54446)

Misc¶

Remove unneeded isort skip comments (#54979)

Doc-only¶

Make term Dag consistent in providers docs (#55101)

1.27.5¶

Release Date: 2025-08-11

Bug Fixes¶

Fix SnowflakeCheckOperator and SnowflakeValueCheckOperator to use parameters arg correctly (#53837)

1.27.4¶

Release Date: 2025-08-02

Misc¶

Fix unreachable code mypy warnings in teradata provider (#53488)
Add sql extras to pandas in the providers where sqlalchemy is needed (#53535)
Add Python 3.13 support for Airflow. (#46891)
Cleanup type ignores (#53297)
Remove type ignore across codebase after mypy upgrade (#53243)
Remove upper-binding for "python-requires" (#52980)
Temporarily switch to use >=,< pattern instead of '~=' (#52967)
Move all BaseHook usages to version_compat in Common SQL (#52800)

1.27.3¶

Release Date: 2025-07-06

Misc¶

Move 'BaseHook' implementation to task SDK (#51873)
Replace models.BaseOperator to Task SDK one for Common Providers (#52443)
Drop support for Python 3.9 (#52072)
Use BaseSensorOperator from task sdk in providers (#52296)
Bump upper binding on pandas in all providers (#52060)

Doc-only¶

Minor pre-commit fixes (#51769)

1.27.2¶

Release Date: 2025-06-18

Bug Fixes¶

fix: Use source hook instead of destination hook when reading records in non-paginated mode in GenericTransfer (#50598)

Misc¶

Port ''ti.run'' to Task SDK execution path (#50141)

1.27.1¶

Release Date: 2025-05-18

Misc¶

Update deprecated functions to use stable functions (#50275)
Update type to follow newest tpying spec (#50229)

1.27.0¶

Release Date: 2025-05-08

Note

This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy.

Bug Fixes¶

Fix SADeprecationWarning when using inspector with SQLAlchemy in DbApiHook (#48938)
Fix: SQLExecuteQueryOperator does not pass extra_dejson values to hook_params (#49282)

Misc¶

Refine type overload for 'common-sql' (#50161)
Bump min Airflow version in providers to 2.10 (#49843)
refactor: Log generated SQL-statement and passed parameters as ERROR message when an exception occurs during insert_rows (#48932)

Doc-only¶

Update standard provider doc operators in core operators-and-hooks-ref.rst (#49401)

1.26.0¶

Release Date: 2025-04-19

Features¶

feat: integrate 'polars' in 'get_df', 'get_df_by_chunks' (#48875)

Misc¶

remove superfluous else block (#49199)

1.25.0¶

Release Date: 2025-04-09

Features¶

Add requires_result_fetch Configuration to SQLExecuteQueryOperator (#46997)

1.24.1¶

Release Date: 2025-03-31

Bug Fixes¶

Fix get_hook() got an unexpected keyword argument 'hook_params' in SQLExecuteQueryTrigger (#47728)
Do not allow semicolon in partition_clause (#48098)

Misc¶

AIP-72: Handle Custom XCom Backend on Task SDK (#47339)
AIP-72: Implement short circuit and branch operators (#46584)

1.24.0¶

Release Date: 2025-03-13

Features¶

Add deferred pagination mode to GenericTransfer (#44809)

Bug Fixes¶

fIx deprecation warnings in common.sql (#47169)
Add missing 'methodtools' dependency of the 'common-sql' provider (#47148)

Misc¶

Upgrade flit to 3.11.0 (#46938)

1.23.0¶

Release Date: 2025-02-26

Note

This version has no code changes. It’s released due to yank of previous version due to packaging issues.

1.22.0¶

Release Date: 2025-02-21

Features¶

Introduce notion of dialects in DbApiHook (#41327)

Bug Fixes¶

Fix DbApiHook.insert_rows logging incorrect number of rows commited (#45789)
Fix escaping of special characters or reserved words as column names in dialects of common sql provider (#45640)

Misc¶

Add run_after column to DagRun model (#45732)
Adding type ignore to sql hook (#46163)
Added ADR document describing why the notion of dialects is introduced (#45456)
Bring back mypy checks for new-structure providers (#45815)
Remove obsolete pandas specfication for pre-python 3.9 (#45399)

1.21.0¶

Release Date: 2024-12-26

Note

This release of provider is only available for Airflow 2.9+ as explained in the Apache Airflow providers support policy.

Warning

All deprecated classes, parameters and features have been removed from the Common SQL provider package. It could be that very old providers versions were depended on private functions of common.sql and as a result users may have created custom code around them without realizing using private functions. This has long been fixed and might affect users of very old provider versions. We are not considering this a breaking change yet we are listing the changes explicitly for visibility. The following breaking changes were introduced:

Hooks
- Remove _make_serializable method from DbApiHook. Use _make_common_data_structure instead.

Features¶

Added output_processor parameter to SQLQueryOperator and fixed bug with return_single_query_results handler when None is passed as split_statements (#44781)
Moved common SQL handler methods of common-sql-provider into dedicated module (#43747)

Misc¶

Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)
Fix static checks in common SQL hooks (#44930)
Added documentation for the SqlThresholdCheckOperator (#44787)
Remove Provider Deprecations in Common SQL (#44645)
Update DAG example links in multiple providers documents (#44034)

1.20.0¶

Release Date: 2024-11-18

Features¶

Add support for semicolon stripping to DbApiHook, PrestoHook, and TrinoHook (#41916)

1.19.0¶

Release Date: 2024-10-31

Features¶

Feature: Added fast_executemany parameter to insert_rows of DbApiHook (#43357)
Make conn id parameters templated in GenericTransfer and also allow passing hook parameters like in BaseSQLOperator (#42891)
Allow capture of replication lag in SqlSensor (#43107)

Misc¶

fix mypy failure (#43206)

1.18.0¶

Release Date: 2024-10-14

Features¶

feat(providers/common/sql): add warning to connection setter (#42736)

Bug Fixes¶

FIX: Only pass connection to sqlalchemy engine in JdbcHook (#42705)

1.17.1¶

Release Date: 2024-10-01

Bug Fixes¶

fix(providers/common/sql): add dummy connection setter for backward compatibility (#42490)
Changed type hinting for handler function (#42275)

1.17.0¶

Release Date: 2024-09-24

Features¶

Note

Connection in DB Hook is now cached to avoid multiple lookups when properties from extras have to be resolved.

Generalize caching of connection in DbApiHook to improve performance (#40751)

Misc¶

feat: log client db messages for provider postgres (#40171)
remove deprecated soft_fail from providers (#41710)

1.16.0¶

Release Date: 2024-08-22

Note

This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy.

Bug Fixes¶

fix: rm deprecated import (#41461)

Misc¶

Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)

1.15.0¶

Release Date: 2024-08-06

Features¶

Create SQLAlchemy engine from connection in DB Hook and added autocommit param to insert_rows method (#40669)

1.14.2¶

Release Date: 2024-07-12

Bug Fixes¶

FIX: DbApiHook.insert_rows unnecessarily restarting connections (#40615)

Misc¶

Enable enforcing pydocstyle rule D213 in ruff. (#40448)

1.14.1¶

Release Date: 2024-06-27

Misc¶

implement per-provider tests with lowest-direct dependency resolution (#39946)
Update pandas minimum requirement for Python 3.12 (#40272)
standardizes template fields for 'BaseSQLOperator' and adds 'database' as a templated field (#39826)

1.14.0¶

Release Date: 2024-05-30

Features¶

Add 'parameters' as template field for SqlSensor (#39588)

Bug Fixes¶

DbAPiHook: Don't log a warning message if placeholder is None and make sure warning message is formatted correctly (#39690)

Misc¶

refactor: The executemany parameter of insert_rows should not be deprecated as for some hooks we don't want to enable a system-wide supports_executemany parameter, that way we can also keep using it in dedicated situations (#39630)
Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)
Add typing for SqlSensor (#39773)

1.13.0¶

Release Date: 2024-05-06

Note

This release of provider is only available for Airflow 2.7+ as explained in the Apache Airflow providers support policy.

Features¶

Add 'sqlalchemy_url' property to 'DbApiHook' class (#38871)
Always use the executemany method when inserting rows in DbApiHook as it's way much faster (#38715)

Bug Fixes¶

Fix 'DbApiHook.insert_rows' when 'rows' is a generator (#38972)
Fix 'update-common-sql-api-stubs' pre-commit check (#38915)

Misc¶

Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)
openlineage, snowflake: do not run external queries for Snowflake (#39113)

1.12.0¶

Release Date: 2024-04-13

Features¶

Add hook_params to template_fields for BaseSQLOperator-related Operators (#38724)
Make 'placeholder' of DbApiHook configurable in UI (#38528)

Misc¶

Undeprecating 'DBApiHookForTests._make_common_data_structure' (#38573)

1.11.1¶

Release Date: 2024-03-08

Bug Fixes¶

Make 'executemany' keyword arguments only in 'DbApiHook.insert_rows' (#37840)
Limit 'pandas' to '<2.2' (#37748)

1.11.0¶

Release Date: 2024-02-18

Features¶

Enhancement: Performance enhancement for insert_rows method DbApiHook with fast executemany + SAP Hana support (#37246)

Bug Fixes¶

Fix SQLThresholdCheckOperator error on falsey vals (#37150)

Misc¶

feat: Switch all class, functions, methods deprecations to decorators (#36876)
Add more-itertools as dependency of common-sql (#37359)

1.10.1¶

Release Date: 2024-01-27

Misc¶

Set min pandas dependency to 1.2.5 for all providers and airflow (#36698)

1.10.0¶

Release Date: 2023-12-27

Make "placeholder" of ODBC configurable in UI (#36000)

Bug Fixes¶

Return common data structure in DBApi derived classes
SQLCheckOperator fails if returns dict with any False values (#36273)

1.9.0¶

Release Date: 2023-12-12

Note

This release of provider is only available for Airflow 2.6+ as explained in the Apache Airflow providers support policy.

Misc¶

Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)
Add Architecture Decision Record for common.sql introduction (#36015)

1.8.1¶

Release Date: 2023-11-29

Misc¶

Add '_make_serializable' method which other SQL operators can overrides when result from cursor is not JSON-serializable (#32319)
Remove backcompat inheritance for DbApiHook (#35754)

1.8.0¶

Release Date: 2023-10-17

Note

This release of provider is only available for Airflow 2.5+ as explained in the Apache Airflow providers support policy.

Bug Fixes¶

fix(providers/sql): respect soft_fail argument when exception is raised (#34199)

Misc¶

Bump min airflow version of providers (#34728)
Use 'airflow.exceptions.AirflowException' in providers (#34511)

1.7.2¶

Release Date: 2023-09-12

Bug Fixes¶

Fix BigQueryValueCheckOperator deferrable mode optimisation (#34018)

Misc¶

Refactor regex in providers (#33898)

1.7.1¶

Release Date: 2023-08-29

Misc¶

Refactor: Better percentage formatting (#33595)
Refactor: Simplify code in smaller providers (#33234)

1.7.0¶

Release Date: 2023-08-14

Features¶

Add a new parameter to SQL operators to specify conn id field (#30784)

1.6.2¶

Release Date: 2023-08-08

Misc¶

Make SQLExecute Query signature consistent with other SQL operators (#32974)
Get rid of Python2 numeric relics (#33050)

1.6.1¶

Release Date: 2023-08-01

Bug Fixes¶

Fix local OpenLineage import in 'SQLExecuteQueryOperator'. (#32400)

Misc¶

Add default port to Openlineage authority method. (#32828)
Add more accurate typing for DbApiHook.run method (#31846)

1.6.0¶

Release Date: 2023-07-09

Features¶

openlineage, common.sql: provide OL SQL parser as internal OpenLineage provider API (#31398)

Misc¶

Pass SQLAlchemy engine to construct information schema query. (#32371)

1.5.2¶

Release Date: 2023-06-23

Misc¶

Remove Python 3.7 support (#30963)

1.5.1¶

Release Date: 2023-05-27

Misc¶

Bring back min-airflow-version for preinstalled providers (#31469)

1.5.0 (YANKED)¶

Warning

This release has been yanked with a reason: This version might cause unconstrained installation of old airflow version lead to Runtime Error.

Note

This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy.

Features¶

Add conditional output processing in SQL operators (#31136)

Misc¶

Remove noisy log from SQL table check (#31037)

1.4.0¶

Release Date: 2023-04-06

Features¶

Add option to show output of 'SQLExecuteQueryOperator' in the log (#29954)

Misc¶

Fix Python API docs formatting for Common SQL provider (#29863)

1.3.4¶

Release Date: 2023-03-06

Bug Fixes¶

Do not process output when do_xcom_push=False (#29599)

1.3.3¶

Release Date: 2023-01-17

Bug Fixes¶

Handle non-compliant behaviour of Exasol cursor (#28744)

1.3.2¶

Release Date: 2023-01-05

Bug Fixes¶

fIx isort problems introduced by recent isort release (#28434)
Fix template rendering for Common SQL operators (#28202)
Defer to hook setting for split_statements in SQLExecuteQueryOperator (#28635)

Misc¶

Clarify docstrings for updated DbApiHook (#27966)

1.3.1¶

Release Date: 2022-11-29

This release fixes a few errors that were introduced in common.sql operator while refactoring common parts:

_process_output method in SQLExecuteQueryOperator has now consistent semantics and typing, it can also modify the returned (and stored in XCom) values in the operators that derive from the SQLExecuteQueryOperator).
descriptions of all returned results are stored as descriptions property in the DBApiHook
last description of the cursor whether to return single query results values are now exposed in DBApiHook via last_description property.

Lack of consistency in the operator caused 1.3.0 to be yanked - the 1.3.0 should not be used - if you have 1.3.0 installed, upgrade to 1.3.1.

Bug Fixes¶

Restore removed (but used) methods in common.sql (#27843)
Fix errors in Databricks SQL operator introduced when refactoring (#27854)
Bump common.sql provider to 1.3.1 (#27888)
Fixing the behaviours of SQL Hooks and Operators finally (#27912)

1.3.0 (YANKED)¶

Warning

This release has been yanked with a reason: Breaks Google 8.4.0 provider for SQLExecute

Note

This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy.

Misc¶

Move min airflow version to 2.3.0 for all providers (#27196)

Features¶

Add SQLExecuteQueryOperator (#25717)
Use DbApiHook.run for DbApiHook.get_records and DbApiHook.get_first (#26944)
DbApiHook consistent insert_rows logging (#26758)

Bug Fixes¶

Common sql bugfixes and improvements (#26761)
Use unused SQLCheckOperator.parameters in SQLCheckOperator.execute. (#27599)

1.2.0¶

Release Date: 2022-09-08

Features¶

Make placeholder style configurable (#25939)
Better error message for pre-common-sql providers (#26051)

Bug Fixes¶

Fix (and test) SQLTableCheckOperator on postgresql (#25821)
Don't use Pandas for SQLTableCheckOperator (#25822)
Discard semicolon stripping in SQL hook (#25855)

1.1.0¶

Release Date: 2022-08-18

Features¶

Improve taskflow type hints with ParamSpec (#25173)
Move all "old" SQL operators to common.sql providers (#25350)
Deprecate hql parameters and synchronize DBApiHook method APIs (#25299)
Unify DbApiHook.run() method with the methods which override it (#23971)
Common SQLCheckOperators Various Functionality Update (#25164)

Bug Fixes¶

Allow Legacy SqlSensor to use the common.sql providers (#25293)
Fix fetch_all_handler & db-api tests for it (#25430)
Align Common SQL provider logo location (#25538)
Fix SQL split string to include ';-less' statements (#25713)

1.0.0¶

Release Date: 2022-07-16

Initial version of the provider. Adds SQLColumnCheckOperator and SQLTableCheckOperator. Moves DBApiHook, SQLSensor and ConnectorProtocol to the provider.