Customizing the image¶
Customizing the image means building it with the Dockerfile that Airflow releases, choosing what goes
into it with --build-arg options instead of adding layers on top of a published image. The quick start
in Step 3: build the image with the extras and packages you need shows the most common case - choosing the extras and adding packages that
need compiling. This page covers everything else you can do with the Dockerfile.
Note
Since Airflow 3.4.0 the images are based on the Docker Hardened Image for Python on Debian 13 “trixie”, see Properties of the hardened base image.
Note
You can usually use the latest Dockerfile released by Airflow to build previous Airflow versions.
Note, however, that there are slight changes in the Dockerfile and entrypoint scripts that can make it
behave slightly differently, depending which Dockerfile version you used. Details of what has changed
in each of the released versions of Docker image can be found in the Changelog.
Prerequisites for building customized docker image:
You need to enable Buildkit to build the image. This can be done by setting
DOCKER_BUILDKIT=1as an environment variable or by installing the buildx plugin and runningdocker buildx buildcommand. Docker Desktop hasBuildkitenabled by default.You need to have a new Docker installed to handle
1.4syntax of the Dockerfile. Docker version23.0.0and above are known to work.
Before attempting to customize the image, you need to download flexible and customizable Dockerfile.
You can extract the officially released version of the Dockerfile from the
released sources.
You can also conveniently download the latest released version
from GitHub. You can save it
in any directory - there is no need for any other files to be present there. If you wish to use your own
files (for example custom configuration of pip or your own requirements or custom dependencies,
you need to use DOCKER_CONTEXT_FILES build arg and place the files in the directory pointed at by
the arg (see Using docker context files for details).
Customizing the image is an optimized way of adding your own dependencies to the image - better
suited to prepare highly optimized (for size) production images, especially when you have dependencies
that require to be compiled before installing (such as mpi4py).
It also allows more sophisticated usages, needed by “Power-users” - for example using forked version of Airflow, or building the images from security-vetted sources.
The big advantage of this method is that it produces optimized image even if you need some compile-time dependencies that are not needed in the final image.
The disadvantage it that building the image takes longer and it requires you to use the Dockerfile that is released as part of Apache Airflow sources.
The disadvantage is that the pattern of building Docker images with --build-arg is less familiar
to developers of such images. However, it is quite well-known to “power-users”. That’s why the
customizing flow is better suited for those users who have more familiarity and have more custom
requirements.
The image also usually builds much longer than the equivalent “Extended” image because instead of extending the layers that are already coming from the base image, it rebuilds the layers needed to add extra dependencies needed at early stages of image building.
When customizing the image you can choose a number of options how you install Airflow:
From the PyPI releases (default)
From the custom installation sources - using additional/replacing the original apt or PyPI repositories
From local sources. This is used mostly during development.
From tag or branch, or specific commit from a GitHub Airflow repository (or fork). This is particularly useful when you build image for a custom version of Airflow that you keep in your fork and you do not want to release the custom Airflow version to PyPI.
From locally stored binary packages for Airflow, Airflow Providers and other dependencies. This is particularly useful if you want to build Airflow in a highly-secure environment where all such packages must be vetted by your security team and stored in your private artifact registry. This also allows to build Airflow image in an air-gapped environment.
Side note. Building
Airflowin anair-gappedenvironment sounds pretty funny, doesn’t it?
You can also add a range of customizations while building the image:
base python image you use for Airflow
version of Airflow to install
extras to install for Airflow (or even removing some default extras)
additional apt/python dependencies to use while building Airflow (DEV dependencies)
add
requirements.txtfile todocker-context-filesdirectory to add extra requirementsadditional apt/python dependencies to install for runtime version of Airflow (RUNTIME dependencies)
additional commands and variables to set if needed during building or preparing Airflow runtime
choosing constraint file to use when installing Airflow
Additional explanation is needed for the last point. Airflow uses constraints to make sure
that it can be predictably installed, even if some new versions of Airflow dependencies are
released (or even dependencies of our dependencies!). The docker image and accompanying scripts
usually determine automatically the right versions of constraints to be used based on the Airflow
version installed and Python version. For example 2.0.2 version of Airflow installed from PyPI
uses constraints from constraints-2.0.2 tag). However, in some cases - when installing Airflow from
GitHub for example - you have to manually specify the version of constraints used, otherwise
it will default to the latest version of the constraints which might not be compatible with the
version of Airflow you use.
You can also download any version of Airflow constraints and adapt it with your own set of constraints and manually set your own versions of dependencies in your own constraints and use the version of constraints that you manually prepared.
You can read more about constraints in Installation from PyPI
Note that if you place requirements.txt in the docker-context-files folder, it will be
used to install all requirements declared there. It is recommended that the file
contains specified version of dependencies to add with == version specifier, to achieve
stable set of requirements, independent if someone releases a newer version. However, you have
to make sure to update those requirements and rebuild the images to account for latest security fixes.
Using docker-context-files¶
When customizing the image, you can optionally make Airflow install custom binaries or provide custom
configuration for your pip in docker-context-files. In order to enable it, you need to add
--build-arg DOCKER_CONTEXT_FILES=docker-context-files build arg when you build the image.
You can pass any subdirectory of your docker context, it will always be mapped to /docker-context-files
during the build.
You can use docker-context-files for the following purposes:
you can place
requirements.txtand add anypippackages you want to install in thedocker-context-filefolder. Those requirements will be automatically installed during the build.
Note
In the example below, we also add apache-airflow package to be installed - in the very same version
that the image version you used it from. This is not strictly necessary, but it is a good practice
to always install the same version of apache-airflow as the one you are using. This way you can
be sure that the version you are using is the same as the one you are extending. In some cases where
your new packages have conflicting dependencies, pip might decide to downgrade or upgrade
apache-airflow for you, so adding it explicitly is a good practice - this way if you have conflicting
requirements, you will get an error message with conflict information, rather than a surprise
downgrade or upgrade of airflow. If you upgrade Airflow base image, you should also update the version
to match the new version of airflow.
/opt/airflow/docker-stack-docs/docker-examples/customizing/own-requirements.sh
mkdir -p docker-context-files
cat <<EOF >./docker-context-files/requirements.txt
beautifulsoup4==4.13.4
apache-airflow==3.0.2
EOF
export DOCKER_BUILDKIT=1
docker build . \
--build-arg DOCKER_CONTEXT_FILES=./docker-context-files \
--tag "my-beautifulsoup4-airflow:0.0.1"
docker run -it my-beautifulsoup4-airflow:0.0.1 python -c 'import bs4; import sys; sys.exit(0)' && \
echo "Success! Beautifulsoup4 installed" && echo
you can place
pip.conf(and legacy.piprc) in thedocker-context-filesfolder and they will be used for allpipcommands (for example you can configure your own sources or authentication mechanisms)
/opt/airflow/docker-stack-docs/docker-examples/customizing/custom-pip.sh
mkdir -p docker-context-files
cat <<EOF >./docker-context-files/pip.conf
[global]
verbose = 2
EOF
export DOCKER_BUILDKIT=1
docker build . \
--build-arg DOCKER_CONTEXT_FILES=./docker-context-files \
--tag "my-custom-pip-verbose-airflow:0.0.1"
you can place
.whlpackages that you downloaded and install them withINSTALL_DISTRIBUTIONS_FROM_CONTEXTset totrue. It’s useful if you build the image in restricted security environments (see: Build images in security restricted environments for details):
/opt/airflow/docker-stack-docs/docker-examples/restricted/restricted_environments.sh
mkdir -p docker-context-files
export AIRFLOW_VERSION="3.5.0"
export BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.14.10-debian13-dev}"
# The packages must be downloaded with the Python version of the base image
[[ "${BASE_IMAGE}" =~ python:([0-9]+\.[0-9]+) ]]
export PYTHON_VERSION="${BASH_REMATCH[1]}"
if [[ "$(python -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')" != "${PYTHON_VERSION}" ]]; then
echo "Run this with Python ${PYTHON_VERSION}, the Python version of ${BASE_IMAGE}"
exit 1
fi
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-${PYTHON_VERSION}.txt" \
"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip download --dest docker-context-files \
--constraint "docker-context-files/constraints-${PYTHON_VERSION}.txt" \
"apache-airflow[async,celery,cncf.kubernetes,elasticsearch,postgres,redis,ssh,statsd]==${AIRFLOW_VERSION}"
Note
You can also pass --build-arg DOCKER_CONTEXT_FILES=. if you want to place your requirements.txt
in the main directory without creating a dedicated folder. However, it is a good practice to keep any files
that you copy to the image context in a sub-folder. This makes it easier to separate things that
are used on the host from those that are passed in Docker context. Of course, by default when you run
docker build . the whole folder is available as “Docker build context” and sent to the docker
engine, but the DOCKER_CONTEXT_FILES are always copied to the build segment of the image so
copying all your local folder might unnecessarily increase time needed to build the image and your
cache will be invalidated every time any of the files in your local folder change.
Warning
BREAKING CHANGE! As of Airflow 2.3.0 you need to specify additional flag:
--build-arg DOCKER_CONTEXT_Files=docker-context-files in order to use the files placed
in docker-context-files. Previously that switch was not needed. Unfortunately this change is needed
in order to enable Dockerfile as standalone Dockerfile without any extra files. As of Airflow 2.3.0
the Dockerfile that is released with Airflow does not need any extra folders or files and can
be copied and used from any folder. Previously you needed to copy Airflow sources together with the
Dockerfile as some scripts were needed to make it work. With Airflow 2.3.0, we are using Buildkit
features that enable us to make the Dockerfile a completely standalone file that can be used “as-is”.
Building from PyPI packages¶
This is the basic way of building the custom images from sources.
The following example builds the production image for the default Python version with latest PyPI-released Airflow, with default set of Airflow extras and dependencies. The latest PyPI-released Airflow constraints are used automatically.
/opt/airflow/docker-stack-docs/docker-examples/customizing/stable-airflow.sh
export DOCKER_BUILDKIT=1
docker build . \
--tag "my-stable-airflow:0.0.1"
The following example builds the production image for the default Python version with default extras from a selected Airflow version. The constraints for that version are used automatically.
/opt/airflow/docker-stack-docs/docker-examples/customizing/pypi-selected-version.sh
export AIRFLOW_VERSION=3.5.0
export DOCKER_BUILDKIT=1
docker build . \
--build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.14.10-debian13-dev}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--tag "my-pypi-selected-version:0.0.1"
The following example builds the production image for the default Python version with additional Airflow extras
(mssql,hdfs) from a selected Airflow version on PyPI, and additional dependency (oauth2client).
/opt/airflow/docker-stack-docs/docker-examples/customizing/pypi-extras-and-deps.sh
export AIRFLOW_VERSION=3.5.0
export DOCKER_BUILDKIT=1
docker build . \
--pull \
--build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.14.10-debian13-dev}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="mssql,hdfs" \
--build-arg ADDITIONAL_PYTHON_DEPS="oauth2client" \
--tag "my-pypi-extras-and-deps:0.0.1"
The following example adds mpi4py package which requires both build-essential and mpi compiler.
/opt/airflow/docker-stack-docs/docker-examples/customizing/add-build-essential-custom.sh
export AIRFLOW_VERSION=3.5.0
export DOCKER_BUILDKIT=1
docker build . \
--pull \
--build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.14.10-debian13-dev}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg ADDITIONAL_PYTHON_DEPS="mpi4py==4.1.1" \
--build-arg ADDITIONAL_DEV_APT_DEPS="libopenmpi-dev" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="openmpi-common" \
--tag "my-build-essential-image:0.0.1"
The above image is equivalent of the “extended” image from previous chapter but its size is only 874 MB. Comparing to 1.1 GB of the “extended image” this is about 230 MB less, so you can achieve ~20% improvement in size of the image by using “customization” vs. extension. The saving can increase in case you have more complex dependencies to build.
Building optimized images¶
The following example builds the production image for the default Python version with additional Airflow extras from PyPI package but it includes additional apt dev and runtime dependencies.
The dev dependencies are those that require build-essential and usually need to involve recompiling
of some python dependencies so those packages might require some additional DEV dependencies to be
present during recompilation. Those packages are not needed at runtime, so we only install them for the
“build” time. They are not installed in the final image, thus producing much smaller images.
In this case pandas requires recompilation so it also needs gcc and g++ as dev APT dependencies.
The jre-headless does not require recompiling so it can be installed as the runtime APT dependency.
/opt/airflow/docker-stack-docs/docker-examples/customizing/pypi-dev-runtime-deps.sh
export AIRFLOW_VERSION=3.5.0
export DOCKER_BUILDKIT=1
docker build . \
--pull \
--build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.14.10-debian13-dev}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="jdbc" \
--build-arg ADDITIONAL_PYTHON_DEPS="pandas==2.1.2" \
--build-arg ADDITIONAL_DEV_APT_DEPS="gcc g++" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="default-jre-headless" \
--tag "my-pypi-dev-runtime:0.0.1"
Building prod images using UV as the package installer¶
The following example builds the production image in default settings, but uses uv to build the image.
This is an experimental feature as uv is a very fast but also very new tool in the Python ecosystem.
/opt/airflow/docker-stack-docs/docker-examples/customizing/use-uv.sh
export DOCKER_BUILDKIT=1
docker build . \
--build-arg AIRFLOW_USE_UV="true" \
--tag "my-custom-use-uv-airflow:0.0.1"
Building images with MySQL client¶
Warning
As of Airflow 3.2.0 - our images only support installation of “MariaDB” client from default Debian
repositories. If you need to use “MySQL” client from Oracle, you need to extend the images and add
your own ways of installing the client. The main reason for that is that Oracle’s
MySQL client installation is broken due to expiring GPG keys and every 2 years the old
packages stop installing unless you manually update the keys (but there is a period of time when the
packages are not installable at all). MariaDB client is a drop-in replacement for MySQL client
and it works perfectly well with MySQL servers.
Using custom installation sources¶
You can customize more aspects of the image - such as additional commands executed before apt dependencies are installed, or adding extra sources to install your dependencies from. You can see all the arguments described below but here is an example of rather complex command to customize the image based on example in this comment:
In case you need to use your custom PyPI package indexes, you can also customize PYPI sources used during
image build by adding a docker-context-files/pip.conf file when building the image.
This pip.conf will not be committed to the repository (it is added to .gitignore) and it will not be
present in the final production image. It is added and used only in the build segment of the image.
Therefore this pip.conf file can safely contain list of package indexes you want to use,
usernames and passwords used for authentication. More details about pip.conf file can be found in the
pip configuration.
If you used the .piprc before (some older versions of pip used it for customization), you can put it
in the docker-context-files/.piprc file and it will be automatically copied to HOME directory
of the airflow user.
Note, that those customizations are only available in the build segment of the Airflow image and they
are not present in the final image. If you wish to extend the final image and add custom .piprc and
pip.conf, you should add them in your own Dockerfile used to extend the Airflow image.
Such customizations are independent of the way how Airflow is installed.
Note
Similar results could be achieved by modifying the Dockerfile manually (see below) and injecting the
commands needed, but by specifying the customizations via build-args, you avoid the need of
synchronizing the changes from future Airflow Dockerfiles. Those customizations should work with the
future version of Airflow’s official Dockerfile at most with minimal modifications od parameter
names (if any), so using the build command for your customizations makes your custom image more
future-proof.
The following - rather complex - example shows capabilities of:
Adding Airflow extras (slack, odbc)
Adding PyPI dependencies (
azure-storage-blob, oauth2client, beautifulsoup4, dateparser, rocketchat_API,typeform)Adding custom environment variables while installing
aptdependencies - both DEV and RUNTIME (ACCEPT_EULA=Y')Adding custom curl command for adding keys and configuring additional apt sources needed to install
aptdependencies (both DEV and RUNTIME)Adding custom
aptdependencies, both DEV (msodbcsql17 unixodbc-dev g++) and runtime msodbcsql17 unixodbc git procps vim)
/opt/airflow/docker-stack-docs/docker-examples/customizing/custom-sources.sh
export AIRFLOW_VERSION=3.5.0
export DOCKER_BUILDKIT=1
docker build . -f Dockerfile \
--pull \
--platform 'linux/amd64' \
--build-arg BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.14.10-debian13-dev}" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg ADDITIONAL_AIRFLOW_EXTRAS="slack,odbc" \
--build-arg ADDITIONAL_PYTHON_DEPS=" \
azure-storage-blob \
oauth2client \
beautifulsoup4 \
dateparser \
rocketchat_API \
typeform" \
--build-arg ADDITIONAL_DEV_APT_COMMAND="curl https://packages.microsoft.com/keys/microsoft.asc | \
apt-key add --no-tty - && \
curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list" \
--build-arg ADDITIONAL_DEV_APT_ENV="ACCEPT_EULA=Y" \
--build-arg ADDITIONAL_DEV_APT_DEPS="msodbcsql17 unixodbc-dev g++" \
--build-arg ADDITIONAL_RUNTIME_APT_COMMAND="curl https://packages.microsoft.com/keys/microsoft.asc | \
apt-key add --no-tty - && \
curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list" \
--build-arg ADDITIONAL_RUNTIME_APT_ENV="ACCEPT_EULA=Y" \
--build-arg ADDITIONAL_RUNTIME_APT_DEPS="msodbcsql17 unixodbc git procps vim" \
--tag "my-custom-sources-image:0.0.1"