Bundles¶

Dag bundles allow Airflow to load Dags from external sources. For a general overview see Dag Bundles.

S3DagBundle¶

Use the S3DagBundle to configure an S3 bundle in your Airflow’s [dag_processor] dag_bundle_config_list.

The prefix selects a subdirectory, with or without a trailing slash. For example, dags and dags/ both download objects under dags/. An empty prefix downloads the whole bucket. IAM policies that restrict listing with the s3:prefix condition must allow the directory prefix including its trailing slash.

Example of using the S3DagBundle:

JSON format example:

export AIRFLOW__DAG_PROCESSOR__DAG_BUNDLE_CONFIG_LIST='[
  {
    "name": "my-s3-dags",
    "classpath": "airflow.providers.amazon.aws.bundles.s3.S3DagBundle",
    "kwargs": {
      "aws_conn_id": "aws_default",
      "bucket_name": "my-airflow-bucket",
      "prefix": "dags/",
      "refresh_interval": 60
    }
  }
]'