Quick start¶
The published images are a good way to try Airflow out, but in production you should run your own image - see Why build your own image. There are three standard ways of building it. Each step below is a little more work than the one before, so take the first one that fits:
Do you only add providers, pure-Python packages, Dags or configuration? Extend the slim image - see Step 1: extend the slim image. This is the right choice for most production images.
Do you need many of the providers of the regular image, or are you trying things out? Extend the regular image - see Step 2: extend the regular image.
Does installing your packages fail with compilation errors, or do you need build tools such as
build-essentialto install them? Build the image from theDockerfilewith the extras and packages you need - see Step 3: build the image with the extras and packages you need. This also gives you the smallest image.
The first two ways are called extending the image, the third one customizing it.
Step 1: extend the slim image¶
Write a Dockerfile that starts FROM the slim image and installs the providers you need. This is
the familiar FROM pattern, needs no Airflow sources and builds in seconds.
/opt/airflow/docker-stack-docs/docker-examples/extending/slim-with-providers/Dockerfile
FROM apache/airflow:slim-3.5.0
RUN PYTHON_MAJOR_MINOR="$(python -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')" \
&& pip install --no-cache-dir \
"apache-airflow==${AIRFLOW_VERSION}" \
apache-airflow-providers-postgres \
apache-airflow-providers-cncf-kubernetes \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_MAJOR_MINOR}.txt"
Two things in this example are worth copying into every image you build:
apache-airflow==${AIRFLOW_VERSION}keeps Airflow at the version of the image. Without it,pipcan silently upgrade or downgrade Airflow to satisfy a dependency of the packages you add; with it, a conflict fails the build with a clear error instead.AIRFLOW_VERSIONis set in the image.The
--constraintfile pins the providers and their dependencies to the versions released and tested with that Airflow version, so the image builds the same way every time. Leave it out only when you want newer versions of the providers than the ones released with your Airflow version.
Then build the image:
docker build . --pull --tag my-company/airflow:0.0.1
Step 2: extend the regular image¶
Extending the regular image works the same way - only the FROM line changes. The providers of the
regular image are already installed, so you only add your own packages:
/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages/Dockerfile
FROM apache/airflow:3.5.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml
The image is about twice as big as an extended slim image, but you do not have to know in advance which
providers you will use. Extending the image has the rules to follow when extending an image and
examples of adding apt packages, requirements.txt files, Dags and configuration.
When adding packages fails with compilation errors¶
Many Python packages ship as pre-built binary wheels and install in seconds. Some do not - for your
Python version, your platform, or at all - and pip then tries to build them from sources. The
published images contain no compiler and no development libraries, so that fails with errors such as:
error: command 'gcc' failed: No such file or directory
fatal error: mpi.h: No such file or directory
Error: pg_config executable not found.
Exception: Can not find valid pkg-config name.
error: can't find Rust compiler
They all end with ERROR: Failed building wheel for <package> or
error: subprocess-exited-with-error.
You can fix this while extending the image - install build-essential and the -dev packages of
the libraries as root before you install the package (see
the example). But the compiler and the development libraries,
hundreds of megabytes of them, then stay in your image, although nothing needs them once the package
is built. Customizing the image avoids that: they are installed only in the build stage of the
Dockerfile, and only the built packages end up in the final image.
Step 3: build the image with the extras and packages you need¶
The Dockerfile that Airflow releases builds the reference images, and its build arguments decide
what goes into the image:
AIRFLOW_EXTRAS- the extras, and therefore the providers, to install. Choosing only the extras you need gives an image as small as an extended slim image, with the constraints for those providers applied automatically.ADDITIONAL_PYTHON_DEPS- other Python packages to install.ADDITIONAL_DEV_APT_DEPS-aptpackages needed only to build them, such as the-devpackages of the libraries they compile against. The compiler itself is always present in the build stage.ADDITIONAL_RUNTIME_APT_DEPS-aptpackages the installed packages need when they run.
Download the Dockerfile to an empty directory and run:
/opt/airflow/docker-stack-docs/docker-examples/customizing/selected-extras.sh
export AIRFLOW_VERSION=3.5.0
export DOCKER_BUILDKIT=1
docker build . \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg AIRFLOW_EXTRAS="postgres,cncf-kubernetes" \
--build-arg ADDITIONAL_PYTHON_DEPS="mpi4py==4.1.1" \
--build-arg ADDITIONAL_DEV_APT_DEPS="libopenmpi-dev" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="openmpi-common" \
--tag "my-selected-extras:0.0.1"
The saving can be large: for the mpi4py package used here, the customized image is about 20%
smaller than the same packages added by extending the regular image, and it grows with every package
that needs compiling.
An image equivalent to the published slim image is built the same way, with no extras at all and the constraints without providers:
docker build . \
--build-arg AIRFLOW_VERSION="3.5.0" \
--build-arg AIRFLOW_EXTRAS="" \
--build-arg AIRFLOW_CONSTRAINTS_MODE="constraints-no-providers" \
--tag my-company/airflow:slim-3.5.0
This method needs BuildKit (the default in Docker Desktop and Docker Engine 23.0 and later) and takes longer to build, because it installs Airflow from scratch. All the build arguments are listed in Image build arguments, and Customizing the image shows what else you can do with them.
Extending vs. customizing the image¶
Extending |
Customizing |
|
|---|---|---|
Uses familiar ‘FROM’ pattern of image building |
Yes |
No |
Requires only basic knowledge about images |
Yes |
No |
Builds quickly |
Yes |
No |
Small image when based on slim / minimal extras |
Yes |
Yes |
Small image with dependencies that need compiling |
No |
Yes |
Can build from custom Airflow sources (forks) |
No |
Yes |
Can build on air-gapped system |
No |
Yes |
You can also combine both: a platform team customizes an optimized base image and publishes it in the
company registry, and Dag authors extend it with FROM.
Where to go next¶
Publishing the image - building, testing and using your image in your deployment.
Extending the image - rules for extending the image and more examples.
Customizing the image - everything you can do with the
Dockerfileand its build arguments.Advanced build cases - which approach to follow for custom package sources, restricted, air-gapped or FIPS environments, forks of Airflow, and the properties of the hardened base image.
Image build arguments - all build arguments of the
Dockerfile.Legacy images - the deprecated
legacyimage flavor, which is not published but which you can build yourself.Recipes - recipes for adding tools such as the Google Cloud SDK.
Airflow Summit 2020’s Production Docker Image talk provides more details about the context, architecture and customization/extension methods for the Production Image.