Reference for package extras¶
Airflow has a number of optional “extras” that you can use to add features to your installation when you are installing Airflow. Those extras are a good way for the users to manage their installation, but also they are useful for contributors to airflow when they want to contribute some of the features - including optional integrations of Airflow - via providers.
Warning
Traditionally in Airflow some of the extras used . and _ to separate the parts of the extra name. This was not PEP-685 normalized name and we opted to change it to to - for all our extras, Expecting that PEP-685 will be implemented in full by pip and other tools we change all our extras to use - as separator even if in some cases it will introduce warnings (the warnings are harmless). This is future proof approach. It’s also fully backwards-compatible if you use _ or . in your extras, but we recommend using - as separator in the future.
Here’s the list of all the extra dependencies of Apache Airflow.
Core Airflow extras¶
These are core airflow extras that extend capabilities of core Airflow. They usually do not install provider
packages (with the exception of celery
and cncf.kubernetes
extras), they just install necessary
python dependencies for the provided package.
extra |
install command |
enables |
---|---|---|
aiobotocore |
|
Support for asynchronous (deferrable) operators for Amazon integration |
async |
|
Async worker classes for Gunicorn |
cgroups |
|
Needed To use CgroupTaskRunner |
edge |
|
Connect Edge Workers via HTTP to the scheduler |
github-enterprise |
|
GitHub Enterprise auth backend |
google-auth |
|
Google auth backend |
graphviz |
|
Graphviz renderer for converting DAG to graphical output |
kerberos |
|
Kerberos integration for Kerberized services (Hadoop, Presto, Trino) |
ldap |
|
LDAP authentication for users |
leveldb |
|
Required for use leveldb extra in google provider |
otel |
|
Required for OpenTelemetry metrics |
pandas |
|
Install Pandas library compatible with Airflow |
password |
|
Password authentication for users |
rabbitmq |
|
RabbitMQ support as a Celery backend |
sentry |
|
Sentry service for application logging and monitoring |
s3fs |
|
Support for S3 as Airflow FS |
saml |
|
Support for SAML authentication in Airflow |
standard |
|
Standard hooks and operators |
statsd |
|
Needed by StatsD metrics |
uv |
|
Install uv - fast, Rust-based package installer (experimental) |
cloudpickle |
|
Cloudpickle hooks and operators |
Providers extras¶
These providers extras are simply convenience extras to install provider packages so that you can install the providers with simple command - including provider package and necessary dependencies in single command, which allows PIP to resolve any conflicting dependencies. This is extremely useful for first time installation where you want to repeatably install version of dependencies which are ‘valid’ for both airflow and providers installed.
For example the below command will install:
apache-airflow
apache-airflow-providers-amazon
apache-airflow-providers-google
apache-airflow-providers-apache-spark
with a consistent set of dependencies based on constraint files provided by Airflow Community at the time 3.0.0.dev0 version was released.
pip install apache-airflow[google,amazon,apache-spark]==3.0.0.dev0 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.0.0.dev0/constraints-3.9.txt"
Note, that this will install providers in the versions that were released at the time of Airflow 3.0.0.dev0 release. You can later upgrade those providers manually if you want to use latest versions of the providers.
Apache Software extras¶
These are extras that add dependencies needed for integration with other Apache projects (note that apache.atlas
and
apache.webhdfs
do not have their own providers - they only install additional libraries that can be used in
custom bash/python providers).
extra |
install command |
enables |
---|---|---|
apache-atlas |
|
Apache Atlas |
apache-beam |
|
Apache Beam operators & hooks |
apache-cassandra |
|
Cassandra related operators & hooks |
apache-drill |
|
Drill related operators & hooks |
apache-druid |
|
Druid related operators & hooks |
apache-flink |
|
Flink related operators & hooks |
apache-hdfs |
|
HDFS hooks and operators |
apache-hive |
|
All Hive related operators |
apache-iceberg |
|
Apache Iceberg hooks |
apache-impala |
|
All Impala related operators & hooks |
apache-kafka |
|
All Kafka related operators & hooks |
apache-kylin |
|
All Kylin related operators & hooks |
apache-livy |
|
All Livy related operators, hooks & sensors |
apache-pig |
|
All Pig related operators & hooks |
apache-pinot |
|
All Pinot related hooks |
apache-spark |
|
All Spark related operators & hooks |
apache-webhdfs |
|
HDFS hooks and operators |
External Services extras¶
These are extras that add dependencies needed for integration with external services - either cloud based or on-premises.
extra |
install command |
enables |
---|---|---|
airbyte |
|
Airbyte hooks and operators |
alibaba |
|
Alibaba Cloud |
apprise |
|
Apprise Notification |
amazon |
|
Amazon Web Services |
asana |
|
Asana hooks and operators |
atlassian-jira |
|
Jira hooks and operators |
microsoft-azure |
|
Microsoft Azure |
cloudant |
|
Cloudant hook |
cohere |
|
Cohere hook and operators |
databricks |
|
Databricks hooks and operators |
datadog |
|
Datadog hooks and sensors |
dbt-cloud |
|
dbt Cloud hooks and operators |
dingding |
|
Dingding hooks and sensors |
discord |
|
Discord hooks and sensors |
|
Facebook Social |
|
github |
|
GitHub operators and hook |
|
Google Cloud |
|
hashicorp |
|
Hashicorp Services (Vault) |
openai |
|
Open AI hooks and operators |
opsgenie |
|
OpsGenie hooks and operators |
pagerduty |
|
Pagerduty hook |
pgvector |
|
pgvector operators and hook |
pinecone |
|
Pinecone Operators and Hooks |
qdrant |
|
Qdrant Operators and Hooks |
salesforce |
|
Salesforce hook |
sendgrid |
|
Send email using sendgrid |
segment |
|
Segment hooks and sensors |
slack |
|
Slack hooks and operators |
snowflake |
|
Snowflake hooks and operators |
tableau |
|
Tableau hooks and operators |
tabular |
|
Tabular hooks |
telegram |
|
Telegram hooks and operators |
vertica |
|
Vertica hook support as an Airflow backend |
weaviate |
|
Weaviate hook and operators |
yandex |
|
Yandex.cloud hooks and operators |
ydb |
|
YDB hooks and operators |
zendesk |
|
Zendesk hooks |
Locally installed software extras¶
These are extras that add dependencies needed for integration with other software packages installed usually as part of the deployment of Airflow. Some of those enable Airflow to use executors to run tasks with them - other than via the built-in LocalExecutor.
extra |
install command |
brings |
enables executors |
---|---|---|---|
arangodb |
|
ArangoDB operators, sensors and hook |
|
celery |
|
Celery dependencies and sensor |
CeleryExecutor, CeleryKubernetesExecutor |
cncf-kubernetes |
|
Kubernetes client libraries, KubernetesPodOperator & friends |
KubernetesExecutor, LocalKubernetesExecutor |
docker |
|
Docker hooks and operators |
|
edge |
|
Connect Edge Workers via HTTP to the scheduler |
EdgeExecutor |
elasticsearch |
|
Elasticsearch hooks and Log Handler |
|
exasol |
|
Exasol hooks and operators |
|
fab |
|
FAB auth manager |
|
github |
|
GitHub operators and hook |
|
influxdb |
|
Influxdb operators and hook |
|
jenkins |
|
Jenkins hooks and operators |
|
mongo |
|
Mongo hooks and operators |
|
microsoft-mssql |
|
Microsoft SQL Server operators and hook. |
|
mysql |
|
MySQL operators and hook |
|
neo4j |
|
Neo4j operators and hook |
|
odbc |
|
ODBC data sources including MS SQL Server |
|
openfaas |
|
OpenFaaS hooks |
|
oracle |
|
Oracle hooks and operators |
|
postgres |
|
PostgreSQL operators and hook |
|
presto |
|
All Presto related operators & hooks |
|
redis |
|
Redis hooks and sensors |
|
samba |
|
Samba hooks and operators |
|
singularity |
|
Singularity container operator |
|
teradata |
|
Teradata hooks and operators |
|
trino |
|
All Trino related operators & hooks |
Other extras¶
These are extras that provide support for integration with external systems via some - usually - standard protocols.
The entries with *
in the Preinstalled
column indicate that those extras (providers) are always
pre-installed when Airflow is installed.
extra |
install command |
enables |
Preinstalled |
---|---|---|---|
common-compat |
|
Compatibility code for old Airflow |
|
common-io |
|
Core IO Operators |
|
common-messaging |
|
Core Messaging Operators |
|
common-sql |
|
Core SQL Operators |
|
ftp |
|
FTP hooks and operators |
|
grpc |
|
Grpc hooks and operators |
|
http |
|
HTTP hooks, operators and sensors |
|
imap |
|
IMAP hooks and sensors |
|
jdbc |
|
JDBC hooks and operators |
|
microsoft-psrp |
|
PSRP hooks and operators |
|
microsoft-winrm |
|
WinRM hooks and operators |
|
openlineage |
|
Sending OpenLineage events |
|
opensearch |
|
Opensearch hooks and operators |
|
papermill |
|
Papermill hooks and operators |
|
sftp |
|
SFTP hooks, operators and sensors |
|
smtp |
|
SMTP hooks and operators |
|
sqlite |
|
SQLite hooks and operators |
|
ssh |
|
SSH hooks and operators |
Doc extras¶
Those are the extras that are needed to generated documentation for Airflow. This is used for development time only
extra |
install command |
enables |
---|---|---|
doc |
|
Packages needed to build docs (included in |
doc-gen |
|
Packages needed to generate er diagrams (included in |