Reference for package extras¶
Airflow has a number of optional “extras” that you can use to add features to your installation when you are installing Airflow. Those extras are a good way for the users to manage their installation, but also they are useful for contributors to airflow when they want to contribute some of the features - including optional integrations of Airflow - via providers.
Warning
Traditionally in Airflow some of the extras used . and _ to separate the parts of the extra name. This was not PEP-685 normalized name and we opted to change it to to - for all our extras, Expecting that PEP-685 will be implemented in full by pip and other tools we change all our extras to use - as separator even if in some cases it will introduce warnings (the warnings are harmless). This is future proof approach. It’s also fully backwards-compatible if you use _ or . in your extras, but we recommend using - as separator in the future.
Here’s the list of all the extra dependencies of Apache Airflow.
Core Airflow extras¶
These are core airflow extras that extend capabilities of core Airflow. They usually do not install provider
packages (with the exception of celery
and cncf.kubernetes
extras), they just install necessary
python dependencies for the provided package.
extra |
install command |
enables |
---|---|---|
aiobotocore |
|
Support for asynchronous (deferrable) operators for Amazon integration |
async |
|
Async worker classes for Gunicorn |
cgroups |
|
Needed To use CgroupTaskRunner |
deprecated-api |
|
Deprecated, experimental API that is replaced with the new REST API |
github-enterprise |
|
GitHub Enterprise auth backend |
google-auth |
|
Google auth backend |
graphviz |
|
Graphviz renderer for converting DAG to graphical output |
kerberos |
|
Kerberos integration for Kerberized services (Hadoop, Presto, Trino) |
ldap |
|
LDAP authentication for users |
leveldb |
|
Required for use leveldb extra in google provider |
otel |
|
Required for OpenTelemetry metrics |
pandas |
|
Install Pandas library compatible with Airflow |
password |
|
Password authentication for users |
pydantic |
|
Pydantic serialization for internal-api |
rabbitmq |
|
RabbitMQ support as a Celery backend |
sentry |
|
Sentry service for application logging and monitoring |
s3fs |
|
Support for S3 as Airflow FS |
saml |
|
Support for SAML authentication in Airflow |
statsd |
|
Needed by StatsD metrics |
uv |
|
Install uv - fast, Rust-based package installer (experimental) |
virtualenv |
|
Running python tasks in local virtualenv |
cloudpickle |
pip install apache-airflow[cloudpickle] |
Cloudpickle hooks and operators |
Providers extras¶
These providers extras are simply convenience extras to install provider packages so that you can install the providers with simple command - including provider package and necessary dependencies in single command, which allows PIP to resolve any conflicting dependencies. This is extremely useful for first time installation where you want to repeatably install version of dependencies which are ‘valid’ for both airflow and providers installed.
For example the below command will install:
apache-airflow
apache-airflow-providers-amazon
apache-airflow-providers-google
apache-airflow-providers-apache-spark
with a consistent set of dependencies based on constraint files provided by Airflow Community at the time 2.10.0.dev0 version was released.
pip install apache-airflow[google,amazon,apache-spark]==2.10.0.dev0 \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.10.0.dev0/constraints-3.8.txt"
Note, that this will install providers in the versions that were released at the time of Airflow 2.10.0.dev0 release. You can later upgrade those providers manually if you want to use latest versions of the providers.
Apache Software extras¶
These are extras that add dependencies needed for integration with other Apache projects (note that apache.atlas
and
apache.webhdfs
do not have their own providers - they only install additional libraries that can be used in
custom bash/python providers).
extra |
install command |
enables |
---|---|---|
apache-atlas |
|
Apache Atlas |
apache-beam |
|
Apache Beam operators & hooks |
apache-cassandra |
|
Cassandra related operators & hooks |
apache-drill |
|
Drill related operators & hooks |
apache-druid |
|
Druid related operators & hooks |
apache-flink |
|
Flink related operators & hooks |
apache-hdfs |
|
HDFS hooks and operators |
apache-hive |
|
All Hive related operators |
apache-iceberg |
|
Apache Iceberg hooks |
apache-impala |
|
All Impala related operators & hooks |
apache-kafka |
|
All Kafka related operators & hooks |
apache-kylin |
|
All Kylin related operators & hooks |
apache-livy |
|
All Livy related operators, hooks & sensors |
apache-pig |
|
All Pig related operators & hooks |
apache-pinot |
|
All Pinot related hooks |
apache-spark |
|
All Spark related operators & hooks |
apache-webhdfs |
|
HDFS hooks and operators |
External Services extras¶
These are extras that add dependencies needed for integration with external services - either cloud based or on-premises.
extra |
install command |
enables |
---|---|---|
airbyte |
|
Airbyte hooks and operators |
alibaba |
|
Alibaba Cloud |
apprise |
|
Apprise Notification |
amazon |
|
Amazon Web Services |
asana |
|
Asana hooks and operators |
atlassian-jira |
|
Jira hooks and operators |
microsoft-azure |
|
Microsoft Azure |
cloudant |
|
Cloudant hook |
cohere |
|
Cohere hook and operators |
databricks |
|
Databricks hooks and operators |
datadog |
|
Datadog hooks and sensors |
dbt-cloud |
|
dbt Cloud hooks and operators |
dingding |
|
Dingding hooks and sensors |
discord |
|
Discord hooks and sensors |
|
Facebook Social |
|
github |
|
GitHub operators and hook |
|
Google Cloud |
|
hashicorp |
|
Hashicorp Services (Vault) |
openai |
|
Open AI hooks and operators |
opsgenie |
|
OpsGenie hooks and operators |
pagerduty |
|
Pagerduty hook |
pgvector |
|
pgvector operators and hook |
pinecone |
|
Pinecone Operators and Hooks |
qdrant |
|
Qdrant Operators and Hooks |
salesforce |
|
Salesforce hook |
sendgrid |
|
Send email using sendgrid |
segment |
|
Segment hooks and sensors |
slack |
|
Slack hooks and operators |
snowflake |
|
Snowflake hooks and operators |
tableau |
|
Tableau hooks and operators |
tabular |
|
Tabular hooks |
telegram |
|
Telegram hooks and operators |
vertica |
|
Vertica hook support as an Airflow backend |
weaviate |
|
Weaviate hook and operators |
yandex |
|
Yandex.cloud hooks and operators |
ydb |
|
YDB hooks and operators |
zendesk |
|
Zendesk hooks |
Locally installed software extras¶
These are extras that add dependencies needed for integration with other software packages installed usually as part of the deployment of Airflow. Some of those enable Airflow to use executors to run tasks with them - other than via the built-in LocalExecutor.
extra |
install command |
brings |
enables executors |
---|---|---|---|
arangodb |
|
ArangoDB operators, sensors and hook |
|
celery |
|
Celery dependencies and sensor |
CeleryExecutor, CeleryKubernetesExecutor |
cncf-kubernetes |
|
Kubernetes client libraries, KubernetesPodOperator & friends |
KubernetesExecutor, LocalKubernetesExecutor |
docker |
|
Docker hooks and operators |
|
elasticsearch |
|
Elasticsearch hooks and Log Handler |
|
exasol |
|
Exasol hooks and operators |
|
fab |
|
FAB auth manager |
|
github |
|
GitHub operators and hook |
|
influxdb |
|
Influxdb operators and hook |
|
jenkins |
|
Jenkins hooks and operators |
|
mongo |
|
Mongo hooks and operators |
|
microsoft-mssql |
|
Microsoft SQL Server operators and hook. |
|
mysql |
|
MySQL operators and hook |
|
neo4j |
|
Neo4j operators and hook |
|
odbc |
|
ODBC data sources including MS SQL Server |
|
openfaas |
|
OpenFaaS hooks |
|
oracle |
|
Oracle hooks and operators |
|
postgres |
|
PostgreSQL operators and hook |
|
presto |
|
All Presto related operators & hooks |
|
redis |
|
Redis hooks and sensors |
|
samba |
|
Samba hooks and operators |
|
singularity |
|
Singularity container operator |
|
teradata |
|
Teradata hooks and operators |
|
trino |
|
All Trino related operators & hooks |
Other extras¶
These are extras that provide support for integration with external systems via some - usually - standard protocols.
The entries with *
in the Preinstalled
column indicate that those extras (providers) are always
pre-installed when Airflow is installed.
extra |
install command |
enables |
Preinstalled |
---|---|---|---|
common-compat |
|
Compatibility code for old Airflow |
|
common-io |
|
Core IO Operators |
|
common-sql |
|
Core SQL Operators |
|
ftp |
|
FTP hooks and operators |
|
grpc |
|
Grpc hooks and operators |
|
http |
|
HTTP hooks, operators and sensors |
|
imap |
|
IMAP hooks and sensors |
|
jdbc |
|
JDBC hooks and operators |
|
microsoft-psrp |
|
PSRP hooks and operators |
|
microsoft-winrm |
|
WinRM hooks and operators |
|
openlineage |
|
Sending OpenLineage events |
|
opensearch |
|
Opensearch hooks and operators |
|
papermill |
|
Papermill hooks and operators |
|
sftp |
|
SFTP hooks, operators and sensors |
|
smtp |
|
SMTP hooks and operators |
|
sqlite |
|
SQLite hooks and operators |
|
ssh |
|
SSH hooks and operators |
Production Bundle extras¶
These are extras that install one or more extras as a bundle.
extra |
install command |
enables |
---|---|---|
all |
|
All Airflow user facing features (no devel and doc requirements) |
all-core |
|
All core airflow features that do not require installing providers |
all-dbs |
|
All database integrations |
Development extras¶
The devel
extras only make sense in editable mode. Users of Airflow should not be using them, unless they
start contributing back and install airflow from sources. Those extras are only available in Airflow when
it is installed in editable mode from sources (pip install -e .[devel,EXTRAS]
).
Devel extras¶
The devel extras do not install dependencies for features of Airflow, but add functionality that is needed to develop Airflow, such as running tests, static checks.
extra |
install command |
enables |
---|---|---|
devel-debuggers |
pip install -e ‘.[devel-debuggers]’ |
Adds all test libraries needed to test debuggers |
devel-devscripts |
pip install -e ‘.[devel-devscripts]’ |
Adds all test libraries needed to test devel scripts |
devel-duckdb |
pip install -e ‘.[devel-duckdb]’ |
Adds all test libraries needed to test duckdb |
devel-iceberg |
pip install -e ‘.[devel-iceberg]’ |
Adds all test libraries needed to test iceberg |
devel-mypy |
pip install -e ‘.[devel-mypy]’ |
Adds all test libraries needed to test mypy |
devel-sentry |
pip install -e ‘.[devel-sentry]’ |
Adds all test libraries needed to test sentry |
devel-static-checks |
pip install -e ‘.[devel-static-checks]’ |
Adds all test libraries needed to test static_checks |
devel-tests |
pip install -e ‘.[devel-tests]’ |
Adds all test libraries needed to test tests |
Bundle devel extras¶
Those are extras that bundle devel, editable and doc extras together to make it easy to install them together in a single installation. Some of the extras are more difficult to install on certain systems (such as ARM MacBooks) because they require system level dependencies to be installed.
Note that pip install -e ".[devel]"
should be run at least once, the first time you initialize the editable environment in order
to get minimal, complete test environment with usual tools and dependencies needed for unit testing.
extra |
install command |
enables |
---|---|---|
devel |
|
Minimum development dependencies - minimal, complete test environment |
devel-hadoop |
|
Adds Hadoop stack libraries |
devel-all-dbs |
|
Adds all libraries needed to test database providers |
devel-all |
|
Everything needed for development including Hadoop, all devel extras, all doc extras. Generally: all possible dependencies except providers |
devel-ci |
|
All dependencies required for CI tests (same as |
Doc extras¶
Those are the extras that are needed to generated documentation for Airflow. This is used for development time only
extra |
install command |
enables |
---|---|---|
doc |
|
Packages needed to build docs (included in |
doc-gen |
|
Packages needed to generate er diagrams (included in |
Deprecated 1.10 extras¶
These are the extras that have been deprecated in 2.0 and will be removed in Airflow 3.0.0. They were all replaced by new extras, which have naming consistent with the names of provider packages.
The crypto
extra is not needed any more, because all crypto dependencies are part of airflow package,
so there is no replacement for crypto
extra.
Deprecated extra |
Extra to be used instead |
---|---|
atlas |
apache-atlas |
aws |
amazon |
azure |
microsoft-azure |
cassandra |
apache-cassandra |
crypto |
|
druid |
apache-druid |
gcp |
|
gcp-api |
|
hdfs |
apache-hdfs |
hive |
apache-hive |
kubernetes |
cncf-kubernetes |
mssql |
microsoft-mssql |
pinot |
apache-pinot |
s3 |
amazon |
spark |
apache-spark |
webhdfs |
apache-webhdfs |
winrm |
microsoft-winrm |