Apache Kafka Connection¶
The Apache Kafka connection type configures a connection to Apache Kafka via the confluent-kafka Python package.
Default Connection IDs¶
Kafka hooks and operators use kafka_default by default, this connection is very minimal and should not be assumed useful for more than the most trivial of testing.
Configuring the Connection¶
Connections are configured as a json serializable string provided to the extra field. The error_cb parameter can be
used to specify a callback function by providing a path to the function. e.g "module.callback_func". A full list
of parameters are described in the
Confluent Kafka python library.
If you are defining the Airflow connection from the Airflow UI, the extra field will be renamed to Config Dict.
Most operators and hooks will check that at the minimum the bootstrap.servers key exists and has a value set to be valid.