• Home
• Authoring and Scheduling

↑↓ Navigate ⏎ Select Esc Close

Authoring and Scheduling

Here you can find detailed documentation about advanced authoring and scheduling Airflow Dags. It’s recommended that you first review the pages in core concepts

Authoring

• Deferrable Operators & Triggers
  • Using Deferrable Operators
  • Writing Deferrable Operators
  • High Availability
  • Difference between Mode=’reschedule’ and Deferrable=True in Sensors
• Serialization
  • Serialization resolution order
  • Airflow Object
• Connections & Hooks
  • Hooks
  • Custom connections
• Loops and mapped tasks
  • Choose a mechanism
  • Use them together
• Mapped tasks
  • Simple mapping
  • Mapping with non-TaskFlow operators
  • Assigning multiple parameters to a non-TaskFlow operator
  • Mapping over a task group
  • Filtering items from a mapped task
  • Transforming expanding data
  • Combining upstream data (aka “zipping”)
  • Concatenating multiple upstreams
  • What data types can be expanded?
  • How do templated fields and mapped arguments interact?
  • Placing limits on mapped tasks
  • Automatically skipping zero-length maps
• Loops
  • Create a loop
  • Read the loop context
  • Pass data between iterations
  • Connect a loop to other tasks
  • Use a loop’s result outside the loop
  • Mapped tasks inside a loop
  • Failures, skips, and retries
  • Iterations and execution history
  • Clear tasks inside a loop
• Asset Definitions
  • What is an “Asset”?
  • What is valid URI?
  • Extra information on assets
  • Security Warnings
  • Creating a task to emit asset events
  • Dependency between @asset, @task, and classic operators
  • Output to multiple assets in one task
  • Dynamic data events emitting and asset creation through AssetAlias
  • Cross-team asset event filtering with producer_teams
  • Asset partitions
• DAG Result
  • Marking the result task
  • Retrieving the result
• Non-Python Task SDKs
  • Dag definition
  • Python Dag with stub TaskHandler
  • Coordinator configuration
  • Implementing a new compiled language SDK

Scheduling

• Cron & Time Intervals
  • Cron Presets
• Time Zones
  • Web UI
  • Concepts
  • Time zone aware Dags
• Asset-Aware Scheduling
  • Quickstart
  • Schedule Dags with assets
  • Multiple assets
  • Gate scheduled runs on asset updates
  • Fetching information from a triggering asset event
  • Event-driven scheduling
  • Advanced asset scheduling with conditional expressions
  • Example Use
  • Combining asset and time-based schedules
  • Controlling DagRun creation per asset event
• Timetables
  • Built-in Timetables
  • Timetables comparisons
• Event-Driven Scheduling
  • Supported triggers for event-driven scheduling
  • Consuming Airflow event telemetry from Kafka
  • Use cases for event-driven Dags

Previous Next

Was this entry helpful?