Bundles¶
Dag bundles allow Airflow to load Dags from external sources. For a general overview see Dag Bundles.
S3DagBundle¶
Use the S3DagBundle to configure an S3 bundle in your Airflow’s
[dag_processor] dag_bundle_config_list.
The prefix selects a subdirectory, with or without a trailing slash. For example,
dags and dags/ both download objects under dags/. An empty prefix downloads
the whole bucket. IAM policies that restrict listing with the s3:prefix condition
must allow the directory prefix including its trailing slash.
Example of using the S3DagBundle:
JSON format example:
export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
{
"name": "my-s3-dags",
"classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
"kwargs": {
"aws_conn_id": "aws_default",
"bucket_name": "my-airflow-bucket",
"prefix": "dags/",
"refresh_interval": 60
}
}
]'
S3ArchiveDagBundle¶
Use S3ArchiveDagBundle when the Dags are published as one
.tar.gz object. S3DagBundle downloads every object under
prefix one at a time. Each object costs a full request round-trip, so bundles made of many small files
can take a long time to stage — a cost paid by every component that stages the bundle, which with ephemeral
workers (e.g. KubernetesExecutor task pods) means on every task start.
S3ArchiveDagBundle stages the bundle from the archive_key object: one HEAD request to detect
changes (unchanged archives are not re-downloaded), one GET to fetch it, then a local unpack and a swap
into place. The archive is the only source of the Dags. Configure this class instead of S3DagBundle;
there is no prefix argument and no second copy to keep in sync.
Archive mode requires Python 3.11.4 or later, because unpacking uses tarfile filter="data"
(added in that release). Constructing S3ArchiveDagBundle on an older interpreter raises
RuntimeError.
export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
{
"name": "my-s3-dags",
"classpath": "airflow.providers.amazon.aws.bundles.s3.S3ArchiveDagBundle",
"kwargs": {
"aws_conn_id": "aws_default",
"bucket_name": "my-airflow-bucket",
"archive_key": "bundle-archives/dags.tar.gz",
"refresh_interval": 60
}
}
]'
Publishing the archive is the responsibility of your deployment process. Members are extracted relative to the bundle root, so pack the contents of the Dags directory rather than the directory itself — for example:
tar -C ./dags -czf dags.tar.gz .
aws s3 cp dags.tar.gz s3://my-airflow-bucket/bundle-archives/dags.tar.gz