Why build your own image

The reference images that the community publishes are a good way to try Airflow out, but in production you should run your own image: one that contains exactly the providers and dependencies your Dags need, built once and versioned. Installing packages every time a container starts is fragile and slow - every container pays for the installation again, and a new release of a package can break the start of your deployment. Keep dynamic installation for quick experiments only.

Airflow has more than 100 community-managed providers (installable via extras) and some of the default extras/providers installed are not used by everyone, sometimes others extras/providers are needed, sometimes (very often actually) you need to add your own custom dependencies, packages or even custom providers. You can see all possible extras in Reference for package extras. The set of extras used in Airflow Production image are available in the Dockerfile.

The standard ways of building your own image need nothing but docker and a short Dockerfile or docker build command, and they produce small images suitable for production:

  • Quick start - choose the right approach and build a small production image in a few steps.

  • Extending the image - the rules and more examples for extending the published images.

  • Customizing the image - building the image with the Dockerfile and its build arguments.

  • Advanced build cases - custom package sources, restricted, air-gapped and FIPS environments, forks of Airflow and the details of the base image, starting with a guide to which approach to follow.

  • Publishing the image - building, testing and using your image in your deployment.

  • Legacy images - the deprecated legacy images, built the way the images were built before Airflow 3.4.0. For the 3.4.x line they are not published, but you can build them yourself.

Note

Creating custom images means that you also need to maintain the automation that re-creates them when the packages you install or Airflow itself are upgraded. In cases when you run pure Python tasks, you can also use the Python Virtualenv functions which install Python dependencies dynamically at runtime.

Was this entry helpful?