Advanced build cases¶
Quick start covers the standard ways of building a small production image. This page helps you find the right approach when they are not enough. Find what you want to do in the list below and follow the link.
What do you want to do?¶
Add providers, Python packages, apt packages, Dags or configuration that install without errors - extend the image, see Extending the image.
Add packages that fail to install with compilation errors, or get the smallest possible image - customize the image with additional dev and runtime
aptdependencies, see Building optimized images.Choose the extras, the Airflow version or the Python version the image is built with - customize the image, see Building from PyPI packages and Image build arguments.
Install from your own PyPI index or apt repositories, or authenticate to them - customize the image with
pip.confor custom installation sources, see Using docker-context-files and Using custom installation sources.Build Airflow from a fork, a branch or a commit - customize the image with the
AIRFLOW_INSTALLATION_METHODandAIRFLOW_VERSION_SPECIFICATIONbuild arguments, see Image build arguments.Build in an air-gapped environment or from packages vetted by your security team - see Build images in security restricted environments.
Build an image that has to run in a FIPS-compliant environment - see Build images in FIPS-compliant environments.
Understand why something behaves differently since Airflow 3.4.0 - for example
aptinstalling a second Python, C extensions failing to compile, or renamed Debian packages - see Properties of the hardened base image and Debian trixie.Keep using the Debian “bookworm” based images with python.org Python - build the deprecated
legacyflavor, see Legacy images.Add tools such as the Google Cloud SDK, the Hadoop stack or Go - see Recipes.
Change something that no build argument covers - modify the
Dockerfile, see Modifying the Dockerfile.
Restricted environments¶
Build images in security restricted environments¶
You can also make sure your image is only built using local constraint file and locally downloaded wheel files. This is often useful in Enterprise environments where the binary files are verified and vetted by the security teams. It is also the most complex way of building the image. You should be an expert of building and using Dockerfiles in order to use it and have to have specific needs of security if you want to follow that route.
The example below builds the production image with packages and constraints used from the local
docker-context-files rather than installed from PyPI or GitHub. It also disables MySQL client
installation as it is using external installation method.
Note that as a prerequisite - you need to have downloaded wheel files. In the example below we
first download such constraint file locally and then use pip download to get the .whl files needed
but in most likely scenario, those wheel files should be copied from an internal repository of such .whl
files. Note that AIRFLOW_VERSION_SPECIFICATION is only there for reference, the Apache Airflow .whl file
in the right version is part of the .whl files downloaded.
Note that ‘pip download’ will only works on Linux host as some of the packages need to be compiled from
sources and you cannot install them providing --platform switch. They also need to be downloaded using
the same python version as the target image.
The pip download might happen in a separate environment. The files can be committed to a separate
binary repository and vetted/verified by the security team and used subsequently to build images
of Airflow when needed on an air-gapped system.
Example of preparing the constraint files and wheel files. Note that mysql dependency is removed
as mysqlclient is installed from Oracle’s apt repository and if you want to add it, you need
to provide this library from your repository if you want to build Airflow image in an “air-gapped” system.
/opt/airflow/docker-stack-docs/docker-examples/restricted/restricted_environments.sh
mkdir -p docker-context-files
export AIRFLOW_VERSION="3.5.0"
export BASE_IMAGE="${BASE_IMAGE:-ghcr.io/apache/airflow/base/python:3.14.10-debian13-dev}"
# The packages must be downloaded with the Python version of the base image
[[ "${BASE_IMAGE}" =~ python:([0-9]+\.[0-9]+) ]]
export PYTHON_VERSION="${BASH_REMATCH[1]}"
if [[ "$(python -c 'import sys; print(f"{sys.version_info[0]}.{sys.version_info[1]}")')" != "${PYTHON_VERSION}" ]]; then
echo "Run this with Python ${PYTHON_VERSION}, the Python version of ${BASE_IMAGE}"
exit 1
fi
rm docker-context-files/*.whl docker-context-files/*.tar.gz docker-context-files/*.txt || true
curl -Lo "docker-context-files/constraints-${PYTHON_VERSION}.txt" \
"https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
pip download --dest docker-context-files \
--constraint "docker-context-files/constraints-${PYTHON_VERSION}.txt" \
"apache-airflow[async,celery,cncf.kubernetes,elasticsearch,postgres,redis,ssh,statsd]==${AIRFLOW_VERSION}"
After this step is finished, your docker-context-files folder will contain all the packages that
are needed to install Airflow from.
Those downloaded packages and constraint file can be pre-vetted by your security team before you attempt to install the image. You can also store those downloaded binary packages in your private artifact registry which allows for the flow where you will download the packages on one machine, submit only new packages for security vetting and only use the new packages when they were vetted.
On a separate (air-gapped) system, all the PyPI packages can be copied to docker-context-files
where you can build the image using the packages downloaded by passing those build args:
INSTALL_DISTRIBUTIONS_FROM_CONTEXT="true"- to use packages present indocker-context-filesAIRFLOW_CONSTRAINTS_LOCATION=/docker-context-files/YOUR_CONSTRAINT_FILE.txt- to downloaded constraint files(Optional)
INSTALL_MYSQL_CLIENT="false"if you do not want to installMySQLclient from the Oracle repositories.(Optional)
INSTALL_MSSQL_CLIENT="false"if you do not want to installMsSQLclient from the Microsoft repositories.(Optional)
INSTALL_POSTGRES_CLIENT="false"if you do not want to installPostgresclient from the Postgres repositories.
Note, that the solution we have for installing python packages from local packages, only solves the problem
of “air-gapped” python installation. The Docker image also downloads apt dependencies and node-modules.
Those types of dependencies are more likely to be available in your “air-gapped” system via transparent
proxies and it should automatically reach out to your private registries. However, in the future the
solution might be applied to both of those installation steps.
You can also use techniques described in the previous chapter to make docker build use your private
apt sources or private PyPI repositories (via .pypirc) available which can be security-vetted.
If you fulfill all the criteria, you can build the image on an air-gapped system by running command similar to the below:
/opt/airflow/docker-stack-docs/docker-examples/restricted/restricted_environments.sh
export DOCKER_BUILDKIT=1
docker build . \
--pull \
--build-arg BASE_IMAGE="${BASE_IMAGE}" \
--build-arg AIRFLOW_INSTALLATION_METHOD="apache-airflow" \
--build-arg AIRFLOW_VERSION="${AIRFLOW_VERSION}" \
--build-arg INSTALL_MYSQL_CLIENT="false" \
--build-arg INSTALL_MSSQL_CLIENT="false" \
--build-arg INSTALL_POSTGRES_CLIENT="true" \
--build-arg DOCKER_CONTEXT_FILES="docker-context-files" \
--build-arg INSTALL_DISTRIBUTIONS_FROM_CONTEXT="true" \
--build-arg AIRFLOW_CONSTRAINTS_LOCATION="/docker-context-files/constraints-${PYTHON_VERSION}.txt" \
--tag airflow-my-restricted-environment:0.0.1
Build images in FIPS-compliant environments¶
Airflow images are based on the Docker Hardened Images for Python, and Docker
publishes FIPS-validated variants of those images. Those variants are only available with a paid
Docker subscription, so they cannot be the default, but you can point the build at one with the
BASE_IMAGE build argument.
docker build . --build-arg BASE_IMAGE="dhi.io/python:3.13.16-debian13-fips-dev" --tag my-image:my-tag
Note
While building on a FIPS-validated base image is necessary for FIPS compliance, it is not sufficient to make the image fully FIPS compliant. There might be other reasons for FIPS incompatibility (for example usage of non-FIPS compliant algorithms in the software installed in the image). You should verify the compliance of the image yourself.
The base image¶
Properties of the hardened base image¶
Since Airflow 3.4.0 the images are built on the Debian 13 “trixie” -dev variant of the
Docker Hardened Image for Python (dhi.io/python, mirrored to
ghcr.io/apache/airflow/base/python) instead of compiling Python on debian:bookworm-slim. Docker
publishes the definitions these images are built from in the
docker-hardened-images/catalog repository
(package/python for the Python build, image/python for the image). The differences from the
previous images that can matter when you extend or run the image are:
Python is patched. Docker applies its own patches on top of the upstream CPython release (
package/python/patchin the catalog). One of them makespkgutil.get_data()reject resource paths that are absolute or contain..(CVE-2026-3479) - upstream CPython only documents that restriction and does not enforce it. Libraries that load bundled data through../paths fail withValueError: resource must be a relative path with no parent directory components; for examplemotobefore 5.2.2.Threads get a 1 MiB stack. The hardened Python is compiled with
-DTHREAD_STACK_SIZE=0x100000, while the previous images used theglibcdefault (the 8 MiBulimit -s). On Python 3.12 and 3.13 the interpreter’s C recursion guard is a fixed depth that assumes the larger stack, so deeply nested data - for example a JSON body parsed in a web server worker thread - can crash the process instead of raisingRecursionError. The image restores the 8 MiB default for threads started by Python with aairflow-thread-stack-size.pthfile in the base Python’ssite-packages; delete that file if you need the smaller stacks.Python is built without profile-guided optimization. The previous images were built with
--enable-optimizationsand--with-lto; the hardened Python uses only--with-lto, so CPU-bound Python code can run measurably slower.Python lives under
/usr. The Debian 13 hardened images install Python as Debian packages (python-3.13,libpython-3.13, …), so it is/usr/bin/python3with the standard library in/usr/lib/python3.13(the Debian 12 ones used/opt/python)./usr/pythonand the/usr/local/binlinks point to it, so the paths used before keep working. The standard library ships without.pycfiles, so the build compiles it into the image.The OS is minimal. The base ships no compiler,
curl,wget,git,gzip,whichorldconfig, nodash(/bin/shisbash) and a stripped/etc- for example without/etc/shellsor the PAMcommon-*files. The build restores what Airflow and the Debian packages it installs need; a custom image that relied on something else being present has to install it.Packages also come from Docker’s Debian repository. Next to Debian’s own repositories, the Debian 13 images configure
http://dhi.io/deb/debian/main(/etc/apt/sources.list.d/dhi.sources, signed with/usr/share/keyrings/dhi-deb-main.gpg), which serves Docker’s rebuilds of Debian packages. Their versions carry a+dhi<N>suffix, so apt prefers them over Debian’s builds - in the image and in images extending it. The repository can be used anonymously: unlike pulling thedhi.ioimages, installing packages from it needs nodocker loginor any other credentials.Installing apt packages can pull in a second Python. The Debian 13 images register their own Python with
dpkg(python-3.12,libpython-3.12, …), and Docker’s package repository serves the other Python versions as packages too. A Debian package that depends onpython3- directly or, aslibenchant-2-devdoes, through its build tools - therefore makes apt install another hardened Python version (for examplepython-3.13into a Python 3.12 image), and on the Debian 12 images it pulls in Debian’s ownpython3. The second interpreter does not replace the image’s Python, but it can take over/usr/bin/python3and its headers, and C extensions built afterwards then compile against the wrong Python. The image build fails when this happens, but a custom image that installs extra apt packages is not checked. Check whatapt-get install --simulatewould install before adding packages that might depend onpython3, and prefer the runtime library packages over their-devvariants.The image’s Python packages are held. On the Debian 13 images, Debian’s own
python3packages conflict with the image’spython-3.Xpackages, so a package that apt can only satisfy with them - for example NodeSource’snodejs- would make apt remove the image’s Python without asking. The image marks its Python packages withapt-mark hold, so apt reports thepython3conflict and the installation fails instead. Install that software some other way - for Node.js, copy thenodebinary from the officialnodeimage - rather than releasing the hold.C extensions are compiled more strictly. The Debian 13 images come with GCC 14, which turns incompatible pointer types, integer conversions and implicit function declarations into errors, and the hardened Python passes
-Werror=implicit-function-declarationon to every C extension it builds. Old source distributions that compiled with warnings before - for examplexmlsec1.3.14 oribm-db3.2.0 - fail to build. Prefer versions with wheels or newer releases; if you have to build such a package, settingCFLAGS="-Wno-error=incompatible-pointer-types -Wno-error=int-conversion -Wno-error=implicit-function-declaration"for the installation turns the errors back into warnings.apt does not refresh the shared library cache. The base images ship
libc-binwithout itsdpkgtrigger, so installing a shared library does not runldconfig, and anything that looks libraries up through the cache - such asctypes.util.find_library()- does not find it. The image adds/etc/apt/apt.conf.d/99airflow-ldconfig, which runsldconfigafter everydpkgrun, including those of images extending it.Some OS files are modified.
/etc/os-releaseidentifies the system as “Docker Hardened Images (Debian)”, and/etc/debian_versiondiffers from the Debian package’s copy, so anapt-get upgradethat touchesbase-fileswould stop atdpkg’s interactive configuration-file prompt. The image ships/etc/dpkg/dpkg.cfg.d/airflow-keep-conffiles(force-confdefandforce-confold), sodpkgkeeps the image’s copy of a configuration file without asking - also in images that extend it. A package upgrade therefore does not replace a configuration file that differs from the package’s version.The tags are rebuilt in place, so the default is pinned to a digest. Docker republishes the same tags as CVEs are fixed, and Airflow refreshes its mirror every six hours, so building from a tag alone can pick up a newer base at any time. The default
BASE_IMAGEtherefore names the digest as well, and Airflow’s regular dependency upgrade moves it - in a pull request that has to pass the full CI before the new base is used. Pass a tag without a digest to build on the newest mirrored base instead. The mirror also copies the newest Python patch release Docker publishes, and the same upgrade moves the pinned patch level to it, so the images follow Docker’s releases rather than python.org’s.The OS dependencies come from pinned images too. A digest pins the base image, but not
dhi.io’s apt repository: it moves on - and drops the package versions it replaced - whenever Docker rebuilds its images, so installing the OS dependencies from it can fail for a commit that built fine before. Airflow’s CI and release builds therefore start their OS dependencies - the database clients included - from images in whichinstall_os_dependencies.shalready ran, built daily on the mirrored base images and pinned to a digest by the same upgrade, passed asDEV_OS_DEPS_IMAGEandRUNTIME_OS_DEPS_IMAGE. The script records what it installed with and does nothing when nothing differs, so a build with other apt dependencies or database clients still installs them. A plaindocker buildleaves both atBASE_IMAGEand installs everything as before. TheAIRFLOW_OS_DEPS_IMAGEenvironment variable of an image names the image its OS dependencies came from, and/usr/share/airflow/os-dependencies.base-imagethe base image they were installed on - which can be an older digest thanBASE_IMAGE.Packages come from Docker’s apt repository first. The image keeps stock Debian’s apt sources beside
dhi.io, and/etc/apt/preferences.d/airflow-prefer-dhiranksdhi.iohigher, so apt installs Docker’s build of a package wheneverdhi.iopublishes it and falls back to Debian’s otherwise - also in images that extend it. The priority stays below 1000, so apt still favorsdhi.io’s build but can take a newer version from another repository when a dependency needs it (for example PGDG’slibpq5).
Debian trixie¶
Together with the hardened base, the default images moved from Debian 12 “bookworm” to Debian 13 “trixie”. Besides the newer system libraries, this shows up in two places when you extend the image:
Several library packages were renamed in trixie’s 64-bit
time_ttransition - for examplelibssl3islibssl3t64andlibldap-2.5-0islibldap2- soRUNTIME_APT_DEPSorADDITIONAL_RUNTIME_APT_DEPSthat name the bookworm packages have to be updated.The MySQL-compatible client comes from MariaDB 11.8, the first MariaDB LTS published for trixie. The
mysqlnamed commands are provided bymariadb-client-compat.
The images built before Airflow 3.4.0 - on debian:bookworm-slim, with Python compiled from
python.org sources - can still be built as the deprecated legacy flavor. Those images are not
published; see Legacy images for how to build one.
Recipes¶
Users sometimes share interesting ways of using the Docker images. We encourage users to contribute these recipes to the documentation in case they prove useful to other members of the community by submitting a pull request. The sections below capture this knowledge.
Google Cloud SDK installation¶
Some operators, such as GKEStartPodOperator,
require the installation of Google Cloud SDK (includes gcloud).
You can also run these commands with BashOperator.
Create a new Dockerfile like the one shown below.
/opt/airflow/docker-stack-docs/docker-images-recipes/gcloud.Dockerfile
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
ARG BASE_AIRFLOW_IMAGE
FROM ${BASE_AIRFLOW_IMAGE}
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
USER 0
ARG CLOUD_SDK_VERSION=322.0.0
ENV GCLOUD_HOME=/opt/google-cloud-sdk
ENV PATH="${GCLOUD_HOME}/bin/:${PATH}"
RUN DOWNLOAD_URL="https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-${CLOUD_SDK_VERSION}-linux-x86_64.tar.gz" \
&& TMP_DIR="$(mktemp -d)" \
&& curl -fL "${DOWNLOAD_URL}" --output "${TMP_DIR}/google-cloud-sdk.tar.gz" \
&& mkdir -p "${GCLOUD_HOME}" \
&& tar xzf "${TMP_DIR}/google-cloud-sdk.tar.gz" -C "${GCLOUD_HOME}" --strip-components=1 \
&& "${GCLOUD_HOME}/install.sh" \
--bash-completion=false \
--path-update=false \
--usage-reporting=false \
--additional-components alpha beta kubectl \
--quiet \
&& rm -rf "${TMP_DIR}" \
&& rm -rf "${GCLOUD_HOME}/.install/.backup/" \
&& gcloud --version
USER ${AIRFLOW_UID}
Then build a new image.
docker build . \
--pull \
--build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.2" \
--tag my-airflow-image:0.0.1
Apache Hadoop Stack installation¶
Airflow is often used to run tasks on Hadoop cluster. It required Java Runtime Environment (JRE) to run. Below are the steps to take tools that are frequently used in Hadoop-world:
Java Runtime Environment (JRE)
Apache Hadoop
Apache Hive
Create a new Dockerfile like the one shown below.
/opt/airflow/docker-stack-docs/docker-images-recipes/hadoop.Dockerfile
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
ARG BASE_AIRFLOW_IMAGE
FROM ${BASE_AIRFLOW_IMAGE}
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
USER 0
# Install Java
RUN mkdir -pv /usr/share/man/man1 \
&& mkdir -pv /usr/share/man/man7 \
&& curl -fsSL https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | apt-key add - \
&& echo "deb https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ $(lsb_release -cs) main" > \
/etc/apt/sources.list.d/adoptopenjdk.list \
&& apt-get update \
&& apt-get install --no-install-recommends -y \
adoptopenjdk-8-hotspot-jre \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
ENV JAVA_HOME=/usr/lib/jvm/adoptopenjdk-8-hotspot-jre-amd64
RUN mkdir -p /opt/spark/jars
# Install Apache Hadoop
ARG HADOOP_VERSION=2.10.1
ENV HADOOP_HOME=/opt/hadoop
ENV HADOOP_CONF_DIR=/etc/hadoop
ENV MULTIHOMED_NETWORK=1
ENV USER=root
RUN HADOOP_URL="https://archive.apache.org/dist/hadoop/common/hadoop-$HADOOP_VERSION/hadoop-$HADOOP_VERSION.tar.gz" \
&& curl 'https://dist.apache.org/repos/dist/release/hadoop/common/KEYS' | gpg --import - \
&& curl -fSL "$HADOOP_URL" -o /tmp/hadoop.tar.gz \
&& curl -fSL "$HADOOP_URL.asc" -o /tmp/hadoop.tar.gz.asc \
&& gpg --verify /tmp/hadoop.tar.gz.asc \
&& mkdir -p "${HADOOP_HOME}" \
&& tar -xvf /tmp/hadoop.tar.gz -C "${HADOOP_HOME}" --strip-components=1 \
&& rm /tmp/hadoop.tar.gz /tmp/hadoop.tar.gz.asc \
&& ln -s "${HADOOP_HOME}/etc/hadoop" /etc/hadoop \
&& mkdir "${HADOOP_HOME}/logs" \
&& mkdir /hadoop-data
ENV PATH="$HADOOP_HOME/bin/:$PATH"
# Install Apache Hive
ARG HIVE_VERSION=2.3.7
ENV HIVE_HOME=/opt/hive
ENV HIVE_CONF_DIR=/etc/hive
RUN HIVE_URL="https://archive.apache.org/dist/hive/hive-${HIVE_VERSION}/apache-hive-${HIVE_VERSION}-bin.tar.gz" \
&& curl -fSL 'https://downloads.apache.org/hive/KEYS' | gpg --import - \
&& curl -fSL "$HIVE_URL" -o /tmp/hive.tar.gz \
&& curl -fSL "$HIVE_URL.asc" -o /tmp/hive.tar.gz.asc \
&& gpg --verify /tmp/hive.tar.gz.asc \
&& mkdir -p "${HIVE_HOME}" \
&& tar -xf /tmp/hive.tar.gz -C "${HIVE_HOME}" --strip-components=1 \
&& rm /tmp/hive.tar.gz /tmp/hive.tar.gz.asc \
&& ln -s "${HIVE_HOME}/etc/hive" "${HIVE_CONF_DIR}" \
&& mkdir "${HIVE_HOME}/logs"
ENV PATH="$HIVE_HOME/bin/:$PATH"
# Install GCS connector for Apache Hadoop
# See: https://cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage
ARG GCS_VARIANT="hadoop2"
ARG GCS_VERSION="2.1.5"
RUN GCS_JAR_PATH="/opt/spark/jars/gcs-connector-${GCS_VARIANT}-${GCS_VERSION}.jar" \
&& GCS_JAR_URL="https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-${GCS_VARIANT}-${GCS_VERSION}.jar" \
&& curl "${GCS_JAR_URL}" -o "${GCS_JAR_PATH}"
ENV HADOOP_CLASSPATH="/opt/spark/jars/gcs-connector-${GCS_VARIANT}-${GCS_VERSION}.jar:$HADOOP_CLASSPATH"
USER ${AIRFLOW_UID}
Then build a new image.
docker build . \
--pull \
--build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.0.2" \
--tag my-airflow-image:0.0.1
Apache Beam Go Stack installation¶
To be able to run Beam Go Pipeline with the BeamRunGoPipelineOperator,
you will need Go in your container. Install Airflow with apache-airflow-providers-google>=6.5.0 and apache-airflow-providers-apache-beam>=3.2.0
Create a new Dockerfile like the one shown below.
/opt/airflow/docker-stack-docs/docker-images-recipes/go-beam.Dockerfile
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
ARG BASE_AIRFLOW_IMAGE
FROM ${BASE_AIRFLOW_IMAGE}
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
USER 0
ARG GO_VERSION=1.16.4
ENV GO_INSTALL_DIR=/usr/local/go
# Install Go
RUN if [[ "$(uname -a)" = *"x86_64"* ]] ; then export ARCH=amd64 ; else export ARCH=arm64 ; fi \
&& DOWNLOAD_URL="https://dl.google.com/go/go${GO_VERSION}.linux-${ARCH}.tar.gz" \
&& TMP_DIR="$(mktemp -d)" \
&& curl -fL "${DOWNLOAD_URL}" --output "${TMP_DIR}/go.linux-${ARCH}.tar.gz" \
&& mkdir -p "${GO_INSTALL_DIR}" \
&& tar xzf "${TMP_DIR}/go.linux-${ARCH}.tar.gz" -C "${GO_INSTALL_DIR}" --strip-components=1 \
&& rm -rf "${TMP_DIR}"
ENV GOROOT=/usr/local/go
ENV PATH="$GOROOT/bin:$PATH"
USER ${AIRFLOW_UID}
Then build a new image.
docker build . \
--pull \
--build-arg BASE_AIRFLOW_IMAGE="apache/airflow:2.2.5" \
--tag my-airflow-image:0.0.1
Modifying the Dockerfile¶
The build arg approach is a convenience method if you do not want to manually modify the Dockerfile.
Our approach is flexible enough to be able to accommodate most requirements and
customizations out-of-the-box. When you use it, you do not need to worry about adapting the image every
time a new version of Airflow is released. However, sometimes it is not enough if you have very
specific needs and want to build a very custom image. In such case you can simply modify the
Dockerfile manually as you see fit and store it in your forked repository. However, you will have to
make sure to rebase your changes whenever new version of Airflow is released, because we might modify
the approach of our Dockerfile builds in the future and you might need to resolve conflicts
and rebase your changes.
There are a few things to remember when you modify the Dockerfile:
We are using the widely recommended pattern of
.dockerignorewhere everything is ignored by default and only the required folders are added through exclusion (!). This allows to keep docker context small because there are many binary artifacts generated in the sources of Airflow and if they are added to the context, the time of building the image would increase significantly. If you want to add any new folders to be available in the image you must add them here with leading!# Ignore everything ** # Allow only these directories !airflow ...
The
docker-context-filesfolder is automatically added to the context of the image, so if you want to add individual files, binaries, requirement files etc you can add them there. Thedocker-context-filesis copied to the/docker-context-filesfolder of the build segment of the image, so it is not present in the final image - which makes the final image smaller in case you want to use those files only in thebuildsegment. You must copy any files from the directory manually, using COPY command if you want to get the files in your final image (in the main image segment).
Dependency caching¶
When you customize the image, the Dockerfile keeps the packages that pip and uv download and
build in a BuildKit cache mount. The
cache lives in your builder, not in the image, so it does not make the image bigger - but it makes the
next build faster: even when a change invalidates the layer that installs Python packages, the packages
that did not change are taken from the cache instead of being downloaded or compiled again.
This matters when you build the image repeatedly - for example while you try out different
ADDITIONAL_PYTHON_DEPS, extras or a requirements.txt in docker-context-files, or when the
dependencies you add have to be compiled. It does not help on CI runners that start with an empty builder
every time, unless you keep the builder’s cache between runs.
The cache is identified by two values:
TARGETARCH- set by Docker to the architecture you build for, so packages built for one architecture are never used for another.DEPENDENCY_CACHE_EPOCH- a build argument. Pass a new value to start with an empty cache, for example when you suspect a broken or stale package in it, after changing the system libraries your packages are compiled against, or when you want to check that the image still builds from scratch.
You can also remove all cache mounts of your builder with docker builder prune --filter type=exec.cachemount.
More details¶
Image build arguments - all build arguments of the
Dockerfile.Images documentation - the architecture of the images: their context, parameters and internal structure.