apache-airflow-providers-apache-kafka
Changelog¶
2.0.0¶
Release Date: 2026-09-14
Note
librdkafka callbacks given as dotted-path strings on a Kafka connection extra (error_cb,
throttle_cb, stats_cb, log_cb, oauth_cb, on_commit) are no longer imported
unless the full importable path of the callback is listed in the new [apache_kafka]
callback_allowlist option, which is empty by default. A connection that relies on such a
callback now raises ValueError until its path is added to the allowlist. Callbacks passed as
actual callables, and managed authentication (Amazon MSK IAM, Google Managed Kafka), are unaffected.
Breaking changes¶
Add an allowlist config option for Kafka connection string callbacks (#72208)
Doc-only¶
Improve documentation for the KafkaEventProducer plugin (#71085)
1.16.0¶
Release Date: 2026-08-23
Features¶
Add 'return_apply_function_results' to ConsumeFromTopicOperator to return apply_function results (#69740)
1.15.1¶
Release Date: 2026-08-08
Bug Fixes¶
Fix Kafka consumer not being closed on error in ConsumeFromTopicOperator (#69641)Fix AwaitMessageTrigger crash on tombstone messages when apply_function is unset (#69665)Fix KafkaBaseHook.test_connection missing oauth_cb for managed Kafka (#69507)Fix AwaitMessageTriggerFunctionSensor not honoring timeout (#69639)Validate ProduceToTopicOperator topic after rendering (#70333)
1.15.0¶
Release Date: 2026-07-28
Features¶
Add Kafka Event Producer publishing DagRun and TaskInstance state-change events (#68082, #70014)Add Amazon MSK IAM (OAUTHBEARER) support to Apache Kafka provider (#69427)
Doc-only¶
Document 'apply_function' dot notation requirement for Kafka message queues (#68414)Document each provider's optional extras in its docs index (#69478)
1.14.0¶
Release Date: 2026-05-23
Features¶
Add uri sanitizers and asset factories for new schemes (#66426)
Bug Fixes¶
Fix KafkaError.name() called as property instead of method in create_topic (#65734)Fix AwaitMessageTrigger missing _task_instance attribute (#66302)
1.13.3¶
Release Date: 2026-04-26
Bug Fixes¶
Add cleanup to Kafka AwaitMessageTrigger for consumer management (#64612)
1.13.2¶
Release Date: 2026-04-12
Misc¶
Load hook metadata from YAML without importing Hook class (#63826)
1.13.1¶
Release Date: 2026-03-28
Bug Fixes¶
Fix AwaitMessageSensor not honoring timeout (#62104)Fix docstring for AwaitMessageSensor and AwaitMessageTrigger in apache-kafka-provider (#63333)
Misc¶
Add Python 3.14 Support (#63520)Migrate apache/kafka connection UI metadata to YAML (#63171)
1.13.0¶
Release Date: 2026-03-13
Features¶
Add commit_offset option to AwaitMessageSensor and AwaitMessageTrigger (#62916)
1.12.0¶
Release Date: 2026-02-14
Features¶
Support Python 3.13 in apache.kafka provider (#61434)
1.11.3¶
Release Date: 2026-01-17
Misc¶
Consume ''AirflowOptionalProviderFeatureException'' from compat sdk in providers (#60335)New year means updated Copyright notices (#60344)
1.11.2¶
Release Date: 2026-01-02
Misc¶
Extract shared "module_loading" distribution (#59139)
1.11.1¶
Release Date: 2025-12-13
Misc¶
Add backcompat for exceptions in providers (#58727)
1.11.0¶
Release Date: 2025-11-30
Note
This release of provider is only available for Airflow 2.11+ as explained in the Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_.
Bug Fixes¶
Fix AwaitMessageSensor to accept timeout and soft_fail parameters (#57863) (#58070)
Misc¶
Bump minimum Airflow version in providers to Airflow 2.11.0 (#58612)
1.10.6¶
Release Date: 2025-11-17
Misc¶
Convert all airflow distributions to be compliant with ASF requirements (#58138)
1.10.5¶
Release Date: 2025-10-26
Misc¶
Make 'AwaitMessageTrigger' inherit 'BaseEventTrigger' (#56741)Migrate Apache providers & Elasticsearch to ''common.compat'' (#57016)
Doc-only¶
1.10.4¶
Release Date: 2025-09-22
Bug Fixes¶
Make 'apply_function' optional in 'AwaitMessageTrigger' (#55437)
1.10.3¶
Release Date: 2025-09-09
Misc¶
Refactor Common Queue Interface (#54651)
Doc-only¶
Make term Dag consistent in providers docs (#55101)
1.10.2¶
Release Date: 2025-08-11
Misc¶
Set minimum version for common.messaging to 1.0.3 (#54160)
1.10.1¶
Release Date: 2025-08-02
Bug Fixes¶
Add validation for commit_cadence in Kafka ConsumeFromTopicOperator (#52015)
Misc¶
Add Python 3.13 support for Airflow. (#46891)Cleanup type ignores (#53300)Remove type ignore across codebase after mypy upgrade (#53243)Make kafka provider compatible with mypy 1.16.1 (#53125)Remove upper-binding for "python-requires" (#52980)Temporarily switch to use >=,< pattern instead of '~=' (#52967)Replace BaseHook to Task SDK for apache/kafka (#52784)
1.10.0¶
Release Date: 2025-07-06
Features¶
Add 'KafkaMessageQueueTrigger' for enhanced message queue trigger usability on Kafka queue (#51718)
Misc¶
Move 'BaseHook' implementation to task SDK (#51873)Update BaseOperator imports for Airflow 3.0 compatibility (#52503)Drop support for Python 3.9 (#52072)
1.9.0¶
Release Date: 2025-05-18
Note
This release of provider is only available for Airflow 2.10+ as explained in the Apache Airflow providers support policy <https://github.com/apache/airflow/blob/main/PROVIDERS.rst#minimum-supported-version-of-airflow-for-community-managed-providers>_.
Features¶
AIP-82: Add KafkaMessageQueueProvider (#49938)
Bug Fixes¶
fix: correct invalid arg timeout (#49426)Fix max_messages warning of Kafka ConsumeFromTopicOperator (#48646)
Misc¶
Bump min Airflow version in providers to 2.10 (#49843)
1.8.1¶
Release Date: 2025-04-19
Misc¶
remove superfluous else block (#49199)
1.8.0¶
Release Date: 2025-03-31
Features¶
Create operators for working with Consumer Groups for GCP Apache Kafka (#47056)
Misc¶
Remove google provider dependencies from apache-kafka provider (#47563)Remove exclusions of confluent-kafka (#47240)Limit confluent-kafka temporarily to exclude 2.8.1 (#47204)Add legacy namespace packages to airflow.providers (#47064)
1.7.0¶
Release Date: 2024-12-26
Note
This release of provider is only available for Airflow 2.9+ as explained in the Apache Airflow providers support policy.
Bug Fixes¶
fix 'ConsumeFromTopicOperator' does not fail even if wrong credentials (#44307)
Misc¶
Bump minimum Airflow version in providers to Airflow 2.9.0 (#44956)Update DAG example links in multiple providers documents (#44034)
1.6.1¶
Release Date: 2024-10-14
Bug Fixes¶
remove callable functions parameter from kafka operator template_fields (#42555)
1.6.0¶
Release Date: 2024-08-22
Note
This release of provider is only available for Airflow 2.8+ as explained in the Apache Airflow providers support policy.
Misc¶
Bump minimum Airflow version in providers to Airflow 2.8.0 (#41396)
1.5.0¶
Release Date: 2024-06-27
Features¶
Add 'delete_topic' to 'KafkaAdminClientHook' (#40142)
Bug Fixes¶
Default client in KafkaBaseHook (#40284)
Misc¶
implement per-provider tests with lowest-direct dependency resolution (#39946)
1.4.1¶
Release Date: 2024-05-30
Bug Fixes¶
Group id is mandatory configuration option for confluent_kafka 2.4.0+ (#39559)
Misc¶
Faster 'airflow_version' imports (#39552)Simplify 'airflow_version' imports (#39497)
1.4.0¶
Release Date: 2024-05-06
Note
This release of provider is only available for Airflow 2.7+ as explained in the Apache Airflow providers support policy.
Misc¶
Bump minimum Airflow version in providers to Airflow 2.7.0 (#39240)
1.3.1¶
Release Date: 2023-12-27
Bug Fixes¶
Provide the consumed message to consumer.commit in AwaitMessageTrigger (#36272)Follow BaseHook connection fields method signature in child classes (#36086)
1.3.0¶
Release Date: 2023-12-12
Note
This release of provider is only available for Airflow 2.6+ as explained in the Apache Airflow providers support policy.
Misc¶
Bump minimum Airflow version in providers to Airflow 2.6.0 (#36017)
1.2.0¶
Release Date: 2023-10-17
Note
This release of provider is only available for Airflow 2.5+ as explained in the Apache Airflow providers support policy.
Bug Fixes¶
Fix typos (double words and it's/its) (#33623)Fixes the Kafka provider's max message limit error (#32926) (#33321)
Misc¶
Bump min airflow version of providers (#34728)Use 'airflow.exceptions.AirflowException' in providers (#34511)
1.1.2¶
Release Date: 2023-07-09
Bug Fixes¶
Break AwaitMessageTrigger execution when finding a message with the desired format (#31803)
1.1.1¶
Release Date: 2023-06-23
Note
This release dropped support for Python 3.7
Misc¶
Remove Python 3.7 support (#30963)
1.1.0¶
Release Date: 2023-05-22
Note
This release of provider is only available for Airflow 2.4+ as explained in the Apache Airflow providers support policy.
Misc¶
Bump minimum Airflow version in providers (#30917)
1.0.0¶
Release Date: 2023-04-24
Initial version of the provider.