Extending the image¶
Extending the image means writing a Dockerfile that starts FROM one of the published images and
adds what you need on top of it. It needs no Airflow sources, uses the familiar FROM pattern and
builds in seconds - it is the right choice for most users. The basic flow is shown in
Quick start; this page has the rules to follow and more examples.
If installing your packages fails with compilation errors, extending is not the best fit - see When adding packages fails with compilation errors and Customizing the image.
Rules to follow when extending the image¶
The image runs as the
airflowuser. Switch torootwith theUSERdirective only for commands that need it (such asapt-get), and switch back toairflowafterwards.Python packages are installed in the
~/.localvirtualenv of theairflowuser. Install yours with plainpip install(oruv pip install) as theairflowuser, and always includeapache-airflow==${AIRFLOW_VERSION}. The constraints the image was built with are in${HOME}/constraints.txt- in the slim image those are the constraints without providers.Keep the image small: use
pip install --no-cache-dir(oruvwith--no-cache), clean theaptlists in the sameRUNcommand that installs packages, and follow the Dockerfile best practices.If you create directories that the running container needs to write to, run
umask 0002in the sameRUNcommand. The image can run as an arbitrary user, which always hasGID=0, so such directories have to be group-writable. See arbitrary docker user.The image contains no compiler. If a dependency needs
build-essentialor other build tools, customize the image instead - see When adding packages fails with compilation errors.You do not need Airflow sources to extend the image: put the
Dockerfileand the files it copies (for example your Dags) in a directory and rundocker buildthere.
Note
The Dockerfile does not strictly follow the SemVer approach of Apache
Airflow when it comes to features and backwards compatibility. Occasionally there are changes in the
build process or in the entrypoint of the image that require slight adaptation of your images. Check
the Changelog when you upgrade.
Common cases of extending¶
The examples below start from a regular image, but they work the same way from a slim one.
PyPI packages¶
When adding packages with pip, use the airflow user (the default), not root - installing
pip packages as root fails with an error message. Add apache-airflow==${AIRFLOW_VERSION} for
the reasons explained in the quick start.
/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages/Dockerfile
FROM apache/airflow:3.5.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml
Dependencies in requirements.txt¶
/opt/airflow/docker-stack-docs/docker-examples/extending/add-requirement-packages/Dockerfile
FROM apache/airflow:3.5.0
COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt
/opt/airflow/docker-stack-docs/docker-examples/extending/add-requirement-packages/requirements.txt
lxml
beautifulsoup4
Pin versions in requirements.txt with == so the image does not change when a new version of a
package is released, and rebuild the image regularly to pick up security fixes.
apt packages¶
When adding packages with apt, switch to the root user, and switch back to the airflow user
when the installation is complete. Clean up the apt lists in the same RUN command to keep the
image small.
/opt/airflow/docker-stack-docs/docker-examples/extending/add-apt-packages/Dockerfile
FROM apache/airflow:3.5.0
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
vim \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
USER airflow
Embedding Dags¶
The following example adds test_dag.py to your image in the /opt/airflow/dags folder.
/opt/airflow/docker-stack-docs/docker-examples/extending/embedding-dags/Dockerfile
FROM apache/airflow:3.5.0
COPY --chown=airflow:root test_dag.py /opt/airflow/dags
import datetime
import pendulum
from airflow.models.dag import DAG
from airflow.providers.standard.operators.empty import EmptyOperator
now = pendulum.now(tz="UTC")
now_to_the_hour = (now - datetime.timedelta(0, 0, 0, 0, 0, 3)).replace(minute=0, second=0, microsecond=0)
START_DATE = now_to_the_hour
DAG_NAME = "test_dag_v1"
dag = DAG(
DAG_NAME,
schedule="*/10 * * * *",
default_args={"depends_on_past": True},
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
)
run_this_1 = EmptyOperator(task_id="run_this_1", dag=dag)
run_this_2 = EmptyOperator(task_id="run_this_2", dag=dag)
run_this_2.set_upstream(run_this_1)
run_this_3 = EmptyOperator(task_id="run_this_3", dag=dag)
run_this_3.set_upstream(run_this_2)
Airflow configuration¶
airflow.cfg in $AIRFLOW_HOME holds Airflow’s configuration. You can set any of its options with
an environment variable named AIRFLOW__{SECTION}__{KEY} (note the double underscores).
/opt/airflow/docker-stack-docs/docker-examples/extending/add-airflow-configuration/Dockerfile
FROM apache/airflow:3.5.0
ENV AIRFLOW__CORE__LOAD_EXAMPLES=True
ENV AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=my_conn_string
More extending examples¶
The examples below cover less common cases.
Provider versions¶
The Airflow Providers are released independently of core Airflow and sometimes you might want to upgrade specific providers only to fix some problems or use features available in that provider version. Here is an example of how you can do it
/opt/airflow/docker-stack-docs/docker-examples/extending/custom-providers/Dockerfile
FROM apache/airflow:3.5.0
RUN pip install "apache-airflow==${AIRFLOW_VERSION}" --no-cache-dir apache-airflow-providers-docker==2.5.1
Provider with apt package¶
The following example adds apache-spark airflow-providers which requires both java and
python package from PyPI.
/opt/airflow/docker-stack-docs/docker-examples/extending/add-providers/Dockerfile
FROM apache/airflow:3.5.0
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
openjdk-21-jre-headless \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
USER airflow
ENV JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" apache-airflow-providers-apache-spark==5.1.1
PyPI with constraints¶
The following example adds lxml python package from PyPI to the image with constraints that were
used to install airflow. This allows you to use the version of packages that you know were tested with the
given version of Airflow. You can also use it if you do not want to use potentially newer versions
that were released after the version of Airflow you are using.
/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages-constraints/Dockerfile
FROM apache/airflow:3.5.0
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" lxml --constraint "${HOME}/constraints.txt"
PyPI with uv¶
The following example adds lxml python package from PyPI to the image using uv. This is an
experimental feature as uv is a very fast but also very new tool in the Python ecosystem.
/opt/airflow/docker-stack-docs/docker-examples/extending/add-pypi-packages-uv/Dockerfile
FROM apache/airflow:3.5.0
# The `uv` tools is Rust packaging tool that is much faster than `pip` and other installers
# Support for uv as installation tool is experimental
RUN uv pip install --no-cache "apache-airflow==${AIRFLOW_VERSION}" lxml
Python venv for external tasks¶
The following example creates a separate Python virtualenv inside the image and installs packages
from requirements.txt into it, leaving Airflow’s own Python environment untouched. This is the
typical setup for the
ExternalPythonOperator
(or the @task.external_python decorator), which executes tasks in a pre-existing, immutable
Python environment. Use /opt/airflow/venv/bin/python as the operator’s python argument
to run tasks in this venv.
The venv is created without --system-site-packages, so the packages installed inside it
are isolated from Airflow’s own dependencies.
/opt/airflow/docker-stack-docs/docker-examples/extending/add-python-venv/Dockerfile
FROM apache/airflow:3.5.0
COPY requirements.txt /
RUN python -m venv /opt/airflow/venv \
&& /opt/airflow/venv/bin/pip install --no-cache-dir -r /requirements.txt
/opt/airflow/docker-stack-docs/docker-examples/extending/add-python-venv/requirements.txt
colorama==0.4.0
Writable directory¶
The following example adds a new directory that is supposed to be writable for any arbitrary user running the container.
/opt/airflow/docker-stack-docs/docker-examples/extending/writable-directory/Dockerfile
FROM apache/airflow:3.5.0
RUN umask 0002; \
mkdir -p ~/writeable-directory
Packages requiring compilation¶
The following example adds mpi4py package which requires both build-essential and mpi compiler.
/opt/airflow/docker-stack-docs/docker-examples/extending/add-build-essential-extend/Dockerfile
FROM apache/airflow:3.5.0
USER root
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
build-essential libopenmpi-dev \
&& apt-get autoremove -yqq --purge \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
USER airflow
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" "mpi4py==4.1.1"
The size of this image is ~ 1.1 GB when built, because the compiler and the development libraries stay in it. Customizing the image builds the same image about 20% smaller - see Building from PyPI packages.