apache-airflow-providers-apache-spark

Changelog¶

5.3.2¶

Release Date: 2025-08-02

Misc¶

Deprecate decorators from Core (#53629)
Add Python 3.13 support for Airflow. (#46891)
Cleanup type ignores (#53301)
Remove type ignore across codebase after mypy upgrade (#53243)
Remove upper-binding for "python-requires" (#52980)
Temporarily switch to use >=,< pattern instead of '~=' (#52967)
Replace BaseHook to Task SDK for apache/pyspark (#52842)
Replace 'BaseHook' to Task SDK for 'apache/spark' (#52683)

5.3.1¶

Release Date: 2025-07-06

Misc¶

Move 'BaseHook' implementation to task SDK (#51873)
add: version_compat (#52448)
Drop support for Python 3.9 (#52072)
Replace 'models.BaseOperator' to Task SDK one for Standard Provider (#52292)

Doc-only¶

Cleanup unused args example_pyspark.py (#52492)

5.3.0¶

Release Date: 2025-05-18

Note

This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_.

Features¶

add root parent information to OpenLineage events (#49237)

Misc¶

Bump Pyspark to even higher version (#50308)
Lower bind pyspark and pydruid to relatively new versions (#50205)
Remove AIRFLOW_2_10_PLUS conditions (#49877)
Bump min Airflow version in providers to 2.10 (#49843)

5.2.1¶

Release Date: 2025-04-19

Misc¶

remove superfluous else block (#49199)

5.2.0¶

Release Date: 2025-04-14

Features¶

Add openlineage as Extra dep for Spark provider (#48972)

Misc¶

Make '@task' import from airflow.sdk (#48896)

5.1.1¶

Release Date: 2025-03-31

Features¶

add OpenLineage configuration injection to SparkSubmitOperator (#47508)

5.0.1¶

Release Date: 2025-03-13

Bug Fixes¶

spark on kubernetes removes dependency on Spark Exit code (#46817)

Misc¶

Upgrade flit to 3.11.0 (#46938)

Doc-only¶

Include driver classpath in --jars cmd docstring in spark-submit hook and operator (#45210)

5.0.0¶

Release Date: 2024-12-26

Note

This release of provider is only available for Airflow 2.9+ as explained in the Apache Airflow providers support policy.

Breaking changes¶

Warning

All deprecated classes, parameters and features have been removed from the Apache Spark provider package. The following breaking changes were introduced:

Operators
- Removed _sql() support for SparkSqlOperator. Please use sql attribute instead. _sql was introduced in 2016 and since it was listed as templated field, which is no longer the case, we handled it as public api despite the _ prefix that marked it as private.

Remove deprecated code from apache spark provider (#44567)

Misc¶

Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)
Fix failing mypy check on 'main' (#44191)
spark-submit: replace 'principle' by 'principal' (#44150)
Update DAG example links in multiple providers documents (#44034)

4.11.3¶

Release Date: 2024-11-18

Misc¶

Move python operator to Standard provider (#42081)

4.11.2¶

Release Date: 2024-10-31

Bug Fixes¶

Changed conf property from str to dict in SparkSqlOperator (#42835)

4.11.1¶

Release Date: 2024-10-14

Misc¶

Refactor function resolve_kerberos_principal (#42777)

4.11.0¶

Release Date: 2024-09-24

Features¶

Add kerberos related connection fields(principal, keytab) on SparkSubmitHook (#40757)

4.10.0¶

Release Date: 2024-08-22

Note

This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy.

Misc¶

Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)
Resolve 'AirflowProviderDeprecationWarning' in 'SparkSqlOperator' (#41358)

4.9.0¶

Release Date: 2024-07-25

Features¶

Add 'kubernetes_application_id' to 'SparkSubmitHook' (#40753)

Bug Fixes¶

(fix): spark submit pod name with driver as part of its name(#40732)

4.8.2¶

Release Date: 2024-06-27

Misc¶

implement per-provider tests with lowest-direct dependency resolution (#39946)

4.8.1¶

Release Date: 2024-05-30

Misc¶

Faster 'airflow_version' imports (#39552)
Simplify 'airflow_version' imports (#39497)

4.8.0¶

Release Date: 2024-05-06

Note

This release of provider is only available for Airflow 2.7+ as explained in the Apache Airflow providers support policy.

Bug Fixes¶

Rename SparkSubmitOperator argument queue as yarn_queue (#38852)

Misc¶

Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)

4.7.2¶

Release Date: 2024-04-13

Misc¶

Rename 'SparkSubmitOperator' fields names to comply with templated fields validation (#38051)
Rename 'SparkSqlOperator' fields name to comply with templated fields validation (#38045)

4.7.1¶

Release Date: 2024-01-27

Misc¶

Bump min version for grpcio-status in spark provider (#36662)

4.7.0¶

Release Date: 2024-01-10

change spark connection form and add spark connections docs (#36419)

4.6.0¶

Release Date: 2023-12-27

Features¶

SparkSubmit: Adding propertyfiles option (#36164)
SparkSubmit Connection Extras can be overridden (#36151)

Bug Fixes¶

Follow BaseHook connection fields method signature in child classes (#36086)

4.5.0¶

Release Date: 2023-12-12

Note

This release of provider is only available for Airflow 2.6+ as explained in the Apache Airflow providers support policy.

Misc¶

Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)

4.4.0¶

Release Date: 2023-11-12

Features¶

Add pyspark decorator (#35247)
Add use_krb5ccache option to SparkSubmitOperator (#35331)

4.3.0¶

Release Date: 2023-10-31

Features¶

Add 'use_krb5ccache' option to 'SparkSubmitHook' (#34386)

4.2.0¶

Release Date: 2023-10-17

Note

This release of provider is only available for Airflow 2.5+ as explained in the Apache Airflow providers support policy.

Misc¶

Bump min airflow version of providers (#34728)

4.1.5¶

Release Date: 2023-09-12

Misc¶

Refactor regex in providers (#33898)

4.1.4¶

Release Date: 2023-08-29

Misc¶

Refactor: Simplify code in Apache/Alibaba providers (#33227)

4.1.3¶

Release Date: 2023-08-08

Bug Fixes¶

Validate conn_prefix in extra field for Spark JDBC hook (#32946)

4.1.2¶

Release Date: 2023-08-01

Note

The provider now expects apache-airflow-providers-cncf-kubernetes in version 7.4.0+ installed in order to run Spark on Kubernetes jobs. You can install the provider with cncf.kubernetes extra with pip install apache-airflow-providers-spark[cncf.kubernetes] to get the right version of the cncf.kubernetes provider installed.

Misc¶

Move all k8S classes to cncf.kubernetes provider (#32767)

4.1.1¶

Release Date: 2023-06-23

Note

This release dropped support for Python 3.7

Misc¶

SparkSubmitOperator: rename spark_conn_id to conn_id (#31952)

4.1.0¶

Release Date: 2023-05-22

Note

This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy.

Misc¶

Bump minimum Airflow version in providers (#30917)

4.0.1¶

Release Date: 2023-04-06

Bug Fixes¶

Only restrict spark binary passed via extra (#30213)
Validate host and schema for Spark JDBC Hook (#30223)
Add spark3-submit to list of allowed spark-binary values (#30068)

4.0.0¶

Release Date: 2022-11-18

Note

This release of provider is only available for Airflow 2.3+ as explained in the Apache Airflow providers support policy.

Breaking changes¶

The spark-binary connection extra could be set to any binary, but with 4.0.0 version only two values are allowed for it spark-submit and spark2-submit.

The spark-home connection extra is not allowed anymore - the binary should be available on the PATH in order to use SparkSubmitHook and SparkSubmitOperator.

Remove custom spark home and custom binaries for spark (#27646)

Misc¶

Move min airflow version to 2.3.0 for all providers (#27196)

3.0.0¶

Release Date: 2022-06-13

Breaking changes¶

Note

This release of provider is only available for Airflow 2.2+ as explained in the Apache Airflow providers support policy.

Bug Fixes¶

Add typing for airflow/configuration.py (#23716)
Fix backwards-compatibility introduced by fixing mypy problems (#24230)

Misc¶

AIP-47 - Migrate spark DAGs to new design #22439 (#24210)
chore: Refactoring and Cleaning Apache Providers (#24219)

2.1.3¶

Release Date: 2022-03-26

Bug Fixes¶

Fix mistakenly added install_requires for all providers (#22382)

2.1.2¶

Release Date: 2022-03-19

Misc¶

Add Trove classifiers in PyPI (Framework :: Apache Airflow :: Provider)

2.1.1¶

Release Date: 2022-03-10

Bug Fixes¶

fix param rendering in docs of SparkSubmitHook (#21788)

Misc¶

Support for Python 3.10

2.1.0¶

Release Date: 2022-02-13

Features¶

Add more SQL template fields renderers (#21237)
Add optional features in providers. (#21074)

2.0.3¶

Release Date: 2022-01-06

Bug Fixes¶

Ensure Spark driver response is valid before setting UNKNOWN status (#19978)

2.0.2¶

Release Date: 2021-12-06

Bug Fixes¶

fix bug of SparkSql Operator log going to infinite loop. (#19449)

2.0.1¶

Release Date: 2021-09-03

Misc¶

Optimise connection importing for Airflow 2.2.0

2.0.0¶

Release Date: 2021-06-23

Breaking changes¶

Auto-apply apply_default decorator (#15667)

Warning

Due to apply_default decorator removal, this version of the provider requires Airflow 2.1.0+. If your Airflow version is < 2.1.0, and you want to install this provider version, first upgrade Airflow to at least version 2.1.0. Otherwise your Airflow package version will be upgraded automatically and you will have to manually run airflow upgrade db to complete the migration.

Bug fixes¶

Make SparkSqlHook use Connection (#15794)

1.0.3¶

Release Date: 2021-05-06

Bug fixes¶

Fix 'logging.exception' redundancy (#14823)

1.0.2¶

Release Date: 2021-03-07

Bug fixes¶

Use apache.spark provider without kubernetes (#14187)

1.0.1¶

Release Date: 2021-02-08

Updated documentation and readme files.

1.0.0¶

Release Date: 2020-12-14

Initial version of the provider.