• Home
• Authoring and Scheduling

Authoring and Scheduling

Here you can find detailed documentation about advanced authoring and scheduling airflow DAGs. It’s recommended that you first review the pages in core concepts

Authoring

• Deferrable Operators & Triggers
  • Using Deferrable Operators
  • Writing Deferrable Operators
  • High Availability
  • Difference between Mode=’reschedule’ and Deferrable=True in Sensors
• Serialization
  • Airflow Object
• Connections & Hooks
  • Hooks
  • Custom connections
• Dynamic Task Mapping
  • Simple mapping
  • Mapping with non-TaskFlow operators
  • Assigning multiple parameters to a non-TaskFlow operator
  • Mapping over a task group
  • Filtering items from a mapped task
  • Transforming expanding data
  • Combining upstream data (aka “zipping”)
  • Concatenating multiple upstreams
  • What data types can be expanded?
  • How do templated fields and mapped arguments interact?
  • Placing limits on mapped tasks
  • Automatically skipping zero-length maps

Scheduling

• Cron & Time Intervals
  • Cron Presets
• Time Zones
  • Web UI
  • Concepts
  • Time zone aware DAGs
• Assets
  • Quickstart
  • What is an “Asset”?
  • What is valid URI?
  • Extra information on asset
  • How to use assets in your DAGs
  • Multiple assets
  • Attaching extra information to an emitting asset event
  • Fetching information from previously emitted asset events
  • Fetching information from a triggering asset event
  • Manipulating queued asset events through REST API
  • Advanced asset scheduling with conditional expressions
  • Example Use
  • Dynamic data events emitting and asset creation through AssetAlias
  • Combining asset and time-based schedules
• Timetables
  • Built-in Timetables
  • Timetables comparisons
• Event-driven scheduling
  • Example: Triggering a DAG from an external message queue
  • How it works
  • Supported triggers for event-driven scheduling
  • Use cases for event-driven DAGs

Previous Next

Was this entry helpful?