• Home
• Core Concepts

↑↓ Navigate ⏎ Select Esc Close

Core Concepts

Here you can find detailed documentation about each one of the core concepts of Apache Airflow® and how to use them, as well as a high-level architectural overview.

Architecture

• Architecture Overview
  • Airflow components
  • Deploying Airflow components
  • Architecture Diagrams
  • Workloads
  • Control Flow
  • User interface

Workloads

• Dags
  • Declaring a Dag
  • Loading Dags
  • Running Dags
  • Dag Assignment
  • Default Arguments
  • The Dag decorator
  • Control Flow
  • Dynamic Dags
  • Dag Visualization
  • Dag & Task Documentation
  • Packaging Dags
  • .airflowignore
  • Dag Dependencies
  • Dag pausing, deactivation and deletion
  • Dag Auto-pausing (Experimental)
  • Deadline Alerts
  • Testing a Dag
• Dag Runs
  • Dag Run Status
  • Re-run Dag
  • UI
  • CLI
  • External Triggers
  • To Keep in Mind
• Tasks
  • Relationships
  • Task Instances
  • Timeouts
  • SLAs
  • Special Exceptions
  • Task Instance Heartbeat Timeout
  • Executor Configuration
• Operators
  • Jinja Templating
  • Reserved params keyword
  • Pre- and post-execute methods
  • Difference between pre-/post-execute and setup/teardown
• Sensors
  • BaseSensorOperator parameters
  • Common sensors
• TaskFlow
  • Context
  • Logging
  • Passing Arbitrary Objects As Arguments
  • Sensors and the TaskFlow API
  • History
• Executor
  • Executor Types
  • Using Multiple Executors Concurrently
  • Writing Your Own Executor
• Auth manager
  • Available auth managers to use
  • Why pluggable auth managers?
  • Writing your own auth manager
  • Dag and Dag Sub-Component Authorization
  • Next Steps
• Multi-Team
  • When to Use Multi-Team Mode
  • Core Concepts
  • Enabling Multi-Team Mode
  • Creating and Managing Teams
  • Configuring Team Resources
  • Team-based Executor Configuration
  • Dag Bundle to Team Association
  • How Scheduling Works
  • Architecture
  • Important Considerations
• Object Storage
  • Cloud Object Stores are not real file systems
  • Basic Use
  • Configuration
  • Path API
  • Extensions
  • Copying and Moving
  • External Integrations
• Backfill
  • Control over data reprocessing
  • Concurrency control
  • Run ordering
  • Dry run
  • Example
• Message Queues

Communication

• XComs
  • Object Storage XCom Backend
  • Custom XCom Backends
  • Verifying Custom XCom Backend usage in Containers
• Variables
• Params
  • Dag-level Params
  • Task-level Params
  • Referencing Params in a Task
  • JSON Schema Validation
  • Use Params to Provide a Trigger UI Form
  • Disabling Runtime Param Modification

Debugging

• Debugging Airflow Dags
• Debugging Airflow Dags on the command line

Previous Next

Was this entry helpful?