Edge Worker Deployment¶

Edge Workers can be deployed outside of the central Airflow infrastructure. They are connected to the Airflow API server via HTTP(s). The Edge Worker is a lightweight component that can be deployed on any machine that has outbound HTTP(s) access to the Airflow API server. The Edge Worker is designed to be lightweight and easy to deploy. It allows you to run Airflow tasks on machines that are not part of your main data center, e.g. edge servers. This also allows to deploy only reduced dependencies on the edge worker.

Here are a few imperative requirements for your workers:

airflow needs to be installed, and the Airflow CLI needs to be in the path. This includes the Task SDK as well as the edge3 provider package.
Airflow configuration settings should be homogeneous across the cluster and on the edge site
Operators that are executed on the Edge Worker need to have their dependencies met in that context. Please take a look to the respective provider package documentations
The worker needs to have access to the DAGS_FOLDER, and you need to synchronize the filesystems by your own means. A common setup would be to store your DAGS_FOLDER in a Git repository and sync it across machines using Chef, Puppet, Ansible, or whatever you use to configure machines in your environment. If all your boxes have a common mount point, having your pipelines files shared there should work as well

Minimum Airflow configuration settings for the Edge Worker to make it running is:

Section [core]
- executor: Executor must be set or added to be airflow.providers.edge3.executors.EdgeExecutor
- internal_api_secret_key: An encryption key must be set on api-server and Edge Worker component as shared secret to authenticate traffic. It should be a random string like the fernet key (but preferably not the same).
Section [edge]
- api_enabled: Must be set to true. It is disabled intentionally to not expose API endpoint by default. This is the endpoint the worker connects to. In a future release a dedicated API server can be started.
- api_url: Must be set to the URL which exposes the api endpoint as it is reachable from the worker. Typically this looks like https://your-hostname-and-port/edge_worker/v1/rpcapi.

To kick off a worker, you need to setup Airflow and kick off the worker subcommand

airflow edge worker

Your worker should start picking up tasks as soon as they get fired in its direction. To stop a worker running on a machine you can use:

airflow edge stop

It will try to stop the worker gracefully by sending SIGINT signal to main process as and wait until all running tasks are completed. Also in a console you can use Ctrl-C to stop the worker.

If you want to monitor the remote activity and worker, use the UI plugin which is included in the provider package and install it on the webserver and use the “Admin” - “Edge Worker Hosts” and “Edge Worker Jobs” pages. (Note: The plugin is not ported to Airflow 3.0 web UI at time of writing)

If you want to check status of the worker via CLI you can use the command

airflow edge status

Some caveats:

Tasks can consume resources. Make sure your worker has enough resources to run worker_concurrency tasks
Make sure that the pool_slots of a Tasks matches with the worker_concurrency of the worker. See also Concurrency slot handling.
Queue names are limited to 256 characters

See Modules Management for details on how Python and Airflow manage modules.

Worker Maintenance Mode¶

Sometimes infrastructure needs to be maintained. The Edge Worker provides a maintenance mode to - Stop accepting new tasks - Drain all ongoing work gracefully

Also please note if the worker detects that the Airflow or Edge provider package version is not the same as the one running on the API server, it will stop accepting new tasks and shut down gracefully. This is to prevent running tasks with different versions of the code.

Worker status can be checked via the web UI in the “Admin” - “Edge Worker Hosts” page.

Note

As of time of writing the web UI to see edge jobs and manage workers is not ported to Airflow 3.0. Until this is available you can use the CLI commands as described in Worker Maintenance Management CLI.

Worker maintenance can also be triggered via the CLI command on the machine that runs the worker.

airflow edge maintenance --comments "Some comments for the maintenance" on

This will stop the local worker instance from accepting new tasks and will complete running tasks. If you add the command argument --wait the CLI will wait until all running tasks are completed before return.

If you want to know the status of you local worker while waiting on maintenance you can use the command

airflow edge status

This will show the status of the local worker instance as JSON and the tasks running on it.

The status and maintenance comments will also be shown in the web UI in the “Admin” - “Edge Worker Hosts” page.

The local worker instance can be started to fetch new tasks via the command

airflow edge maintenance off

This will start the worker again and it will start accepting tasks again.

Worker Maintenance Management CLI¶

Besides the CLI command to trigger maintenance on the local worker instance, there are also additional commands to manage the maintenance of all workers in the cluster. These commands can be used to trigger maintenance on all workers in the cluster or to check the status of all workers in the cluster.

These set of commands need database access, and can only be called on the central Airflow instance. The commands are:

airflow edge list-workers: List all workers in the cluster
airflow edge remote-edge-worker-request-maintenance: Request a remote edge worker to enter maintenance mode
airflow edge remote-edge-worker-update-maintenance-comment: Updates the maintenance comment for a remote edge worker
airflow edge remote-edge-worker-exit-maintenance: Request a remote edge worker to exit maintenance mode
airflow edge shutdown-remote-edge-worker: Shuts down a remote edge worker gracefully
airflow edge remove-remote-edge-worker: Remove a worker instance from the cluster