Configuration Reference¶
This page contains the list of all available Airflow configurations for the
apache-airflow-providers-edge provider that can be set in the airflow.cfg file or using environment variables.
Note
The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.
Note
For more information see Setting Configuration Options.
Sections:
[edge]¶
This section only applies if you are using the EdgeExecutor in
[core] section above
api_enabled¶
Flag if the plugin endpoint is enabled to serve Edge Workers.
- Type
boolean
- Default
False- Environment Variable
AIRFLOW__EDGE__API_ENABLED- Example
True
api_url¶
URL endpoint on which the Airflow code edge API is accessible from edge worker.
- Type
string
- Default
None- Environment Variable
AIRFLOW__EDGE__API_URL- Example
https://airflow.hosting.org/edge_worker/v1/rpcapi
heartbeat_interval¶
Edge Worker continuously reports status to the central site. This parameter defines how often a status with heartbeat should be sent. During heartbeat status is reported as well as it is checked if a running task is to be terminated.
- Type
integer
- Default
30- Environment Variable
AIRFLOW__EDGE__HEARTBEAT_INTERVAL- Example
10
job_fail_purge¶
Minutes after which failed jobs for EdgeExecutor are purged from database
- Type
integer
- Default
60- Environment Variable
AIRFLOW__EDGE__JOB_FAIL_PURGE
job_poll_interval¶
Edge Worker currently polls for new jobs via HTTP. This parameter defines the number of seconds it should sleep between polls for new jobs. Job polling only happens if the Edge Worker seeks for new work. Not if busy.
- Type
integer
- Default
5- Environment Variable
AIRFLOW__EDGE__JOB_POLL_INTERVAL- Example
5
job_success_purge¶
Minutes after which successful jobs for EdgeExecutor are purged from database
- Type
integer
- Default
5- Environment Variable
AIRFLOW__EDGE__JOB_SUCCESS_PURGE
push_log_chunk_size¶
Edge Worker uploads log files in chunks. If the log file part which is uploaded exceeds the chunk size it creates a new request. The application gateway can limit the max body size see: https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size A HTTP 413 issue can point to this value to fix the issue. This value must be defined in Bytes.
- Type
integer
- Default
524288- Environment Variable
AIRFLOW__EDGE__PUSH_LOG_CHUNK_SIZE
worker_concurrency¶
The concurrency defines the default max parallel running task instances and can also be set during
start of worker with the airflow edge worker command parameter. The size of the workers
and the resources must support the nature of your tasks. The parameter
works together with the concurrency_slots parameter of a task.
- Type
integer
- Default
8- Environment Variable
AIRFLOW__EDGE__WORKER_CONCURRENCY