Airflow Summit 2025 is coming October 07-09. Register now for early bird ticket!

Apache Kafka Message Queue

Apache Kafka Queue Provider

Implemented by KafkaMessageQueueProvider

The Apache Kafka Queue Provider is a message queue provider that uses Apache Kafka as the underlying message queue system. It allows you to send and receive messages using Kafka topics in your Airflow workflows. The provider supports Kafka topics and provides features for consuming and processing messages from Kafka brokers.

The queue must be matching this regex:

airflow/providers/apache/kafka/queues/kafka.py[source]

QUEUE_REGEXP = r"^kafka://"

Queue URI Format:

kafka://<broker>/<topic_list>

Where:

  • broker: Kafka brokers (hostname:port)

  • topic_list: Comma-separated list of Kafka topics to consume messages from

The queue parameter is used to configure the underlying AwaitMessageTrigger class and passes all kwargs directly to the trigger constructor, if provided. The apply_function kwarg is required.

Topics can also be specified via the Queue URI instead of the topics kwarg. The provider will extract topics from the URI as follows:

airflow/providers/apache/kafka/queues/kafka.py[source]

        # Parse the queue URI
        parsed = urlparse(queue)
        # Extract topics (after host list)
        # parsed.path starts with a '/', so strip it
        raw_topics = parsed.path.lstrip("/")
        topics = raw_topics.split(",") if raw_topics else []

Was this entry helpful?