airflow.providers.celery.executors.celery_executor

CeleryExecutor.

See also

For more information on how the CeleryExecutor works, take a look at the guide: Celery Executor

Module Contents

Classes

CeleryExecutor

CeleryExecutor is recommended for production use of Airflow.

Functions

__getattr__(name)

Attributes

log

CELERY_SEND_ERR_MSG_HEADER

TaskInstanceInCelery

ARG_BROKER_API

ARG_FLOWER_HOSTNAME

ARG_FLOWER_PORT

ARG_FLOWER_CONF

ARG_FLOWER_URL_PREFIX

ARG_FLOWER_BASIC_AUTH

ARG_AUTOSCALE

ARG_QUEUES

ARG_CONCURRENCY

ARG_CELERY_HOSTNAME

ARG_UMASK

ARG_WITHOUT_MINGLE

ARG_WITHOUT_GOSSIP

CELERY_CLI_COMMAND_PATH

CELERY_COMMANDS

airflow.providers.celery.executors.celery_executor.log[source]
airflow.providers.celery.executors.celery_executor.CELERY_SEND_ERR_MSG_HEADER = 'Error sending Celery task'[source]
airflow.providers.celery.executors.celery_executor.TaskInstanceInCelery[source]
airflow.providers.celery.executors.celery_executor.__getattr__(name)[source]
airflow.providers.celery.executors.celery_executor.ARG_BROKER_API[source]
airflow.providers.celery.executors.celery_executor.ARG_FLOWER_HOSTNAME[source]
airflow.providers.celery.executors.celery_executor.ARG_FLOWER_PORT[source]
airflow.providers.celery.executors.celery_executor.ARG_FLOWER_CONF[source]
airflow.providers.celery.executors.celery_executor.ARG_FLOWER_URL_PREFIX[source]
airflow.providers.celery.executors.celery_executor.ARG_FLOWER_BASIC_AUTH[source]
airflow.providers.celery.executors.celery_executor.ARG_AUTOSCALE[source]
airflow.providers.celery.executors.celery_executor.ARG_QUEUES[source]
airflow.providers.celery.executors.celery_executor.ARG_CONCURRENCY[source]
airflow.providers.celery.executors.celery_executor.ARG_CELERY_HOSTNAME[source]
airflow.providers.celery.executors.celery_executor.ARG_UMASK[source]
airflow.providers.celery.executors.celery_executor.ARG_WITHOUT_MINGLE[source]
airflow.providers.celery.executors.celery_executor.ARG_WITHOUT_GOSSIP[source]
airflow.providers.celery.executors.celery_executor.CELERY_CLI_COMMAND_PATH[source]
airflow.providers.celery.executors.celery_executor.CELERY_COMMANDS = ()[source]
class airflow.providers.celery.executors.celery_executor.CeleryExecutor[source]

Bases: airflow.executors.base_executor.BaseExecutor

CeleryExecutor is recommended for production use of Airflow.

It allows distributing the execution of task instances to multiple worker nodes.

Celery is a simple, flexible and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system.

supports_ad_hoc_ti_run: bool = True[source]
supports_sentry: bool = True[source]
start()[source]

Executors may need to get things started.

sync()[source]

Sync will get called periodically by the heartbeat method.

Executors should override this to perform gather statuses.

debug_dump()[source]

Debug dump; called in response to SIGUSR2 by the scheduler.

update_all_task_states()[source]

Update states of the tasks.

change_state(key, state, info=None, remove_running=True)[source]

Change state of the task.

Parameters
update_task_state(key, state, info)[source]

Update state of a single task.

end(synchronous=False)[source]

Wait synchronously for the previously submitted job to complete.

terminate()[source]

Get called when the daemon receives a SIGTERM.

try_adopt_task_instances(tis)[source]

Try to adopt running task instances that have been abandoned by a SchedulerJob dying.

Anything that is not adopted will be cleared by the scheduler (and then become eligible for re-scheduling)

Returns

any TaskInstances that were unable to be adopted

Return type

Sequence[airflow.models.taskinstance.TaskInstance]

cleanup_stuck_queued_tasks(tis)[source]

Handle remnants of tasks that were failed because they were stuck in queued.

Tasks can get stuck in queued. If such a task is detected, it will be marked as UP_FOR_RETRY if the task instance has remaining retries or marked as FAILED if it doesn’t.

Parameters

tis (list[airflow.models.taskinstance.TaskInstance]) – List of Task Instances to clean up

Returns

List of readable task instances for a warning message

Return type

list[str]

static get_cli_commands()[source]

Vends CLI commands to be included in Airflow CLI.

Override this method to expose commands via Airflow CLI to manage this executor. This can be commands to setup/teardown the executor, inspect state, etc. Make sure to choose unique names for those commands, to avoid collisions.

Was this entry helpful?