Troubleshooting

Obscure task failures

Task state changed externally

This message indicates that the task instance’s state does not match the state reported by another component. The message itself does not identify the root cause.

What this message means

Task state can be updated by different Airflow components or by external actions. If the state reported by the executor does not match the state stored for the task instance, Airflow may log a state mismatch.

One common message looks like this:

Executor <executor> reported that the task instance <ti> finished with state <executor_state>, but the task instance's state attribute is <ti_state>.

Check the logs around the time of the state change to determine what caused it.

How to investigate

Start with the task and scheduler logs, then check the worker or infrastructure logs if the cause is not clear.

Check task logs. If there are no logs in the UI, the task likely never started on a worker, or the worker died before it could write logs.
Check scheduler logs around the same timestamp.
Check worker or executor logs, to see whether the task was received, started, or never dispatched.
Check infrastructure logs (container, pod, or host) for OOM, eviction, or restarts. See Process terminated by signal for SIGTERM and SIGKILL.
Check whether a user or an external process changed the task state in the Airflow UI or through the Airflow REST API.

Common causes

Below are some example scenarios where a task’s state may be changed by a component other than the executor:

If a task’s Dag failed to parse on the worker, the scheduler may mark the task as failed. If confirmed, consider increasing core.dagbag_import_timeout and dag_processor.dag_file_processor_timeout.
A task can be retried or marked as failed if it remains queued longer than scheduler.task_queued_timeout. See Task stuck in queued state.
If a task instance’s heartbeat times out, it will be marked failed by the scheduler. See Task stuck in running state.
The task process was killed by the operating system or orchestrator. See Process terminated by signal.
A user or an external process marked the task as successful or failed in the Airflow UI or through the Airflow REST API.

Task stuck in queued state

A task remains queued while it is waiting to be executed. If it stays queued longer than scheduler.task_queued_timeout (default 600 seconds), it may be retried or marked as failed. There will often be no task logs in the UI, because a worker never ran the task.

Here are some of the common causes:

Not enough worker capacity.
The executor is unable to dispatch tasks to workers.
The queued timeout is shorter than how long tasks wait under normal load.

How to troubleshoot:

Check scheduler logs for tasks stuck in queued.
Confirm workers are running and accepting work.
If tasks wait in queue longer than the timeout under normal load, increase scheduler.task_queued_timeout or add worker capacity.

Task stuck in running state

A task may remain running in the UI even though it appears to make no progress. If the task instance stops sending heartbeats, the scheduler detects a task instance heartbeat timeout (formerly called a zombie task) and may mark the task as failed or reschedule it.

Here are some of the common causes:

The worker ran out of memory and was killed. See Out of memory error (OOM).
The worker stopped running or stopped sending heartbeats, for example after a restart, eviction, scale-down, or liveness probe failure.

How to troubleshoot:

Check the task logs. If they stop abruptly, inspect worker and infrastructure logs for the same timestamp.
Check whether the worker is still running and sending heartbeats.
Check infrastructure logs for OOM kills, restarts, evictions, liveness probe failures, or scale-down events.
If the worker is healthy but heartbeat timeouts continue to occur, review scheduler.task_instance_heartbeat_timeout.

Process terminated by signal

Sometimes, Airflow or some adjacent system will kill a task instance’s TaskRunner, causing the task instance to fail.

Below we discuss a few common cases.

Dag run timeout

A dag run timeout can be specified by dagrun_timeout in the dag’s definition. The task process would likely be killed with SIGTERM (exit code -15).

Out of memory error (OOM)

When a task process consumes too much memory for a worker, the best case scenario is it is killed with SIGKILL (exit code -9). Depending on configuration and infrastructure, it is also possible that the whole worker will be killed due to OOM and then the tasks would be marked as failed after failing to heartbeat.

Lingering task supervisor processes

Under very high concurrency the socket handlers inside the task supervisor may miss the final EOF events from the task process. When this occurs the supervisor believes sockets are still open and will not exit. The workers.socket_cleanup_timeout option controls how long the supervisor waits after the task finishes before force-closing any remaining sockets. If you observe leftover supervisor processes, consider increasing this delay.