Extending the image

Extending the image means writing a Dockerfile that starts FROM one of the published images and adds what you need on top of it. It needs no Airflow sources, uses the familiar FROM pattern and builds in seconds - it is the right choice for most users. The basic flow is shown in Quick start; this page has the rules to follow and more examples.

If installing your packages fails with compilation errors, extending is not the best fit - see When adding packages fails with compilation errors and Customizing the image.

Rules to follow when extending the image

  • The image runs as the airflow user. Switch to root with the USER directive only for commands that need it (such as apt-get), and switch back to airflow afterwards.

  • Python packages are installed in the ~/.local virtualenv of the airflow user. Install yours with plain pip install (or uv pip install) as the airflow user, and always include apache-airflow==${AIRFLOW_VERSION}. The constraints the image was built with are in ${HOME}/constraints.txt - in the slim image those are the constraints without providers.

  • Keep the image small: use pip install --no-cache-dir (or uv with --no-cache), clean the apt lists in the same RUN command that installs packages, and follow the Dockerfile best practices.

  • If you create directories that the running container needs to write to, run umask 0002 in the same RUN command. The image can run as an arbitrary user, which always has GID=0, so such directories have to be group-writable. See arbitrary docker user.

  • The image contains no compiler. If a dependency needs build-essential or other build tools, customize the image instead - see When adding packages fails with compilation errors.

  • You do not need Airflow sources to extend the image: put the Dockerfile and the files it copies (for example your Dags) in a directory and run docker build there.

Note

The Dockerfile does not strictly follow the SemVer approach of Apache Airflow when it comes to features and backwards compatibility. Occasionally there are changes in the build process or in the entrypoint of the image that require slight adaptation of your images. Check the Changelog when you upgrade.

Common cases of extending

The examples below start from a regular image, but they work the same way from a slim one.

PyPI packages

When adding packages with pip, use the airflow user (the default), not root - installing pip packages as root fails with an error message. Add apache-airflow==${AIRFLOW_VERSION} for the reasons explained in the quick start.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages/Dockerfile

FROM apache/airflow:3.5.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml

Dependencies in requirements.txt

/opt/airflow/docker-stack-docs/docker-examples/extending/add-requirement-packages/Dockerfile

FROM apache/airflow:3.5.0
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

/opt/airflow/docker-stack-docs/docker-examples/extending/add-requirement-packages/requirements.txt

lxml
beautifulsoup4

Pin versions in requirements.txt with == so the image does not change when a new version of a package is released, and rebuild the image regularly to pick up security fixes.

apt packages

When adding packages with apt, switch to the root user, and switch back to the airflow user when the installation is complete. Clean up the apt lists in the same RUN command to keep the image small.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-apt-packages/Dockerfile

FROM apache/airflow:3.5.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         vim \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow

Embedding Dags

The following example adds test_dag.py to your image in the /opt/airflow/dags folder.

/opt/airflow/docker-stack-docs/docker-examples/extending/embedding-dags/Dockerfile

FROM apache/airflow:3.5.0

COPY --chown=airflow:root test_dag.py /opt/airflow/dags

docker-examples/extending/embedding-dags/test_dag.py[source]

import datetime

import pendulum

from airflow.models.dag import DAG
from airflow.providers.standard.operators.empty import EmptyOperator

now = pendulum.now(tz="UTC")
now_to_the_hour = (now - datetime.timedelta(0, 0, 0, 0, 0, 3)).replace(minute=0, second=0, microsecond=0)
START_DATE = now_to_the_hour
DAG_NAME = "test_dag_v1"

dag = DAG(
    DAG_NAME,
    schedule="*/10 * * * *",
    default_args={"depends_on_past": True},
    start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
    catchup=False,
)

run_this_1 = EmptyOperator(task_id="run_this_1", dag=dag)
run_this_2 = EmptyOperator(task_id="run_this_2", dag=dag)
run_this_2.set_upstream(run_this_1)
run_this_3 = EmptyOperator(task_id="run_this_3", dag=dag)
run_this_3.set_upstream(run_this_2)

Airflow configuration

airflow.cfg in $AIRFLOW_HOME holds Airflow’s configuration. You can set any of its options with an environment variable named AIRFLOW__{SECTION}__{KEY} (note the double underscores).

/opt/airflow/docker-stack-docs/docker-examples/extending/add-airflow-configuration/Dockerfile

FROM apache/airflow:3.5.0
ENV AIRFLOW__CORE__LOAD_EXAMPLES=True
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=my_conn_string

More extending examples

The examples below cover less common cases.

Provider versions

The Airflow Providers are released independently of core Airflow and sometimes you might want to upgrade specific providers only to fix some problems or use features available in that provider version. Here is an example of how you can do it

/opt/airflow/docker-stack-docs/docker-examples/extending/custom-providers/Dockerfile

FROM apache/airflow:3.5.0
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir apache-airflow-providers-docker==2.5.1

Provider with apt package

The following example adds apache-spark airflow-providers which requires both java and python package from PyPI.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-providers/Dockerfile

FROM apache/airflow:3.5.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         openjdk-21-jre-headless \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
ENV JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" apache-airflow-providers-apache-spark==5.1.1

PyPI with constraints

The following example adds lxml python package from PyPI to the image with constraints that were used to install airflow. This allows you to use the version of packages that you know were tested with the given version of Airflow. You can also use it if you do not want to use potentially newer versions that were released after the version of Airflow you are using.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages-constraints/Dockerfile

FROM apache/airflow:3.5.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml --constraint "${HOME}/constraints.txt"

PyPI with uv

The following example adds lxml python package from PyPI to the image using uv. This is an experimental feature as uv is a very fast but also very new tool in the Python ecosystem.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages-uv/Dockerfile

FROM apache/airflow:3.5.0

# The `uv` tools is Rust packaging tool that is much faster than `pip` and other installers
# Support for uv as installation tool is experimental

RUN uv pip install --no-cache "apache-airflow==${AIRFLOW_VERSION}" lxml

Python venv for external tasks

The following example creates a separate Python virtualenv inside the image and installs packages from requirements.txt into it, leaving Airflow’s own Python environment untouched. This is the typical setup for the ExternalPythonOperator (or the @task.external_python decorator), which executes tasks in a pre-existing, immutable Python environment. Use /opt/airflow/venv/bin/python as the operator’s python argument to run tasks in this venv.

The venv is created without --system-site-packages, so the packages installed inside it are isolated from Airflow’s own dependencies.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-python-venv/Dockerfile

FROM apache/airflow:3.5.0
COPY requirements.txt /
RUN python -m venv /opt/airflow/venv \
    && /opt/airflow/venv/bin/pip install --no-cache-dir -r /requirements.txt

/opt/airflow/docker-stack-docs/docker-examples/extending/add-python-venv/requirements.txt

colorama==0.4.0

Writable directory

The following example adds a new directory that is supposed to be writable for any arbitrary user running the container.

/opt/airflow/docker-stack-docs/docker-examples/extending/writable-directory/Dockerfile

FROM apache/airflow:3.5.0
RUN umask 0002; \
    mkdir -p ~/writeable-directory

Packages requiring compilation

The following example adds mpi4py package which requires both build-essential and mpi compiler.

/opt/airflow/docker-stack-docs/docker-examples/extending/add-build-essential-extend/Dockerfile

FROM apache/airflow:3.5.0
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         build-essential libopenmpi-dev \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" "mpi4py==4.1.1"

The size of this image is ~ 1.1 GB when built, because the compiler and the development libraries stay in it. Customizing the image builds the same image about 20% smaller - see Building from PyPI packages.

Was this entry helpful?