Checking Airflow Health Status

Airflow has two methods to check the health of components - HTTP checks and CLI checks. All available checks are accessible through the CLI, but only some are accessible through HTTP due to the role of the component being checked and the tools being used to monitor the deployment.

For example, when running on Kubernetes, use a Liveness probes (livenessProbe property) with CLI checks on the scheduler deployment to restart it when it fails. For the webserver, you can configure the readiness probe (readinessProbe property) using Webserver Health Check Endpoint.

For an example for a Docker Compose environment, see the docker-compose.yaml file available in the Running Airflow in Docker.

Webserver Health Check Endpoint

To check the health status of your Airflow instance, you can simply access the endpoint /api/v2/monitor/health. It will return a JSON object that provides a high-level glance at the health status across multiple Airflow components, including per-instance details when multiple schedulers, triggerers, or Dag processors are running.

{
  "metadatabase": {
    "status": "healthy"
  },
  "scheduler": {
    "status": "healthy",
    "latest_scheduler_heartbeat": "2018-12-26T17:15:11+00:00",
    "detailed_status": "healthy",
    "instances": [
      {
        "status": "healthy",
        "hostname": "scheduler-1.example.com",
        "latest_scheduler_heartbeat": "2018-12-26T17:15:11+00:00"
      }
    ]
  },
  "triggerer": {
    "status": "healthy",
    "latest_triggerer_heartbeat": "2018-12-26T17:16:12+00:00",
    "detailed_status": "degraded",
    "instances": [
      {
        "status": "healthy",
        "hostname": "triggerer-1.example.com",
        "latest_triggerer_heartbeat": "2018-12-26T17:16:12+00:00",
        "team_name": "team-a"
      },
      {
        "status": "unhealthy",
        "hostname": "triggerer-2.example.com",
        "latest_triggerer_heartbeat": "2018-12-26T17:10:00+00:00",
        "team_name": null
      }
    ]
  },
  "dag_processor": {
    "status": "healthy",
    "latest_dag_processor_heartbeat": "2018-12-26T17:16:12+00:00",
    "detailed_status": "healthy",
    "instances": [
      {
        "status": "healthy",
        "hostname": "dag-processor-1.example.com",
        "latest_dag_processor_heartbeat": "2018-12-26T17:16:12+00:00",
        "bundle_names": ["dags-team-a"]
      }
    ]
  }
}

metadatabase
- status is "healthy" when a valid connection can be initiated with the database, otherwise "unhealthy".
Component-level fields for scheduler, triggerer, and dag_processor
- status (legacy aggregate): "healthy" if any running instance is alive, otherwise "unhealthy" (including when no running jobs exist for that component).
- detailed_status: reflects the full set of running instances:
  - "healthy" — every running instance is alive
  - "degraded" — some instances are alive and some are not
  - "down" — no running instance is alive (including when no jobs exist)
- latest_*_heartbeat: the most recent heartbeat among running jobs of that type (ordered by heartbeat descending), or null when there are none. An instance is considered alive when its latest heartbeat is within the component health-check threshold (defaults and config options: [scheduler] scheduler_health_check_threshold, [triggerer] triggerer_health_check_threshold, [dag_processor] health_check_threshold).
- instances: one entry per running job of that type (null when there are none). Each entry includes:
  - status: "healthy" or "unhealthy" for that instance
  - hostname: host where the component is running
  - the corresponding latest_*_heartbeat for that instance
  - team_name (triggerer only): team the triggerer is scoped to, or null when unscoped
  - bundle_names (Dag processor only): Dag bundles that processor is configured to parse, or null when unset
- For HA deployments, prefer detailed_status and instances when you need to see every scheduler, triggerer, or Dag processor. The top-level status remains useful for simple probes that only care whether at least one instance is healthy.

Please keep in mind that the HTTP response code of /api/v2/monitor/health endpoint should not be used to determine the health status of the application. The return code is only indicative of the state of the rest call (200 for success).

Served by the web server, this health check endpoint is independent of the newer Scheduler Health Check Server, which optionally runs on each scheduler.

Note

For this check to work, at least one working web server is required. Suppose you use this check for scheduler monitoring, then in case of failure of the web server, you will lose the ability to monitor scheduler, which means that it can be restarted even if it is in good condition. For greater confidence, consider using CLI Check for Scheduler or Scheduler Health Check Server.
Using this endpoint as webserver probes (liveness/readiness) makes it contingent on Airflow core components’ availability (database, scheduler, etc). Webservers will be frequently restarted if any of these core components are down. To make Webservers less prone to other components’ failures, consider using endpoints like api/v2/version.

Scheduler Health Check Server

In order to check scheduler health independent of the web server, Airflow optionally starts a small HTTP server in each scheduler to serve a scheduler /health endpoint. It returns status code 200 when the scheduler is healthy and status code 503 when the scheduler is unhealthy. To run this server in each scheduler, set [scheduler]enable_health_check to True. By default, it is False. The server is running on the port specified by the [scheduler]scheduler_health_check_server_port option. By default, it is 8974. We are using http.server.BaseHTTPRequestHandler as a small server.

CLI Check for Scheduler

Scheduler creates an entry in the table airflow.jobs.job.Job with information about the host and timestamp (heartbeat) at startup, and then updates it regularly. You can use this to check if the scheduler is working correctly. To do this, you can use the airflow jobs check command. On failure, the command will exit with a non-zero error code.

To check if the local scheduler is still working properly, run:

airflow jobs check --job-type SchedulerJob --local

To check if any scheduler is running when you are using high availability, run:

airflow jobs check --job-type SchedulerJob --allow-multiple --limit 100

CLI Check for Database

To verify that the database is working correctly, you can use the airflow db check command. On failure, the command will exit with a non-zero error code.

HTTP monitoring for Celery Cluster

You can optionally use Flower to monitor the health of the Celery cluster. It also provides an HTTP API that you can use to build a health check for your environment.

For details about installation, see: Celery Executor. For details about usage, see: The Flower project documentation.

CLI Check for Celery Workers

To verify that the Celery workers are working correctly, you can use the celery inspect ping command. On failure, the command will exit with a non-zero error code.

Note

For this check to work, [celery]worker_enable_remote_control must be True. If the parameter is set to False, the command will exit with a non-zero error code.

To check if the worker running on the local host is working correctly, run:

celery --app airflow.providers.celery.executors.celery_executor.app inspect ping -d celery@${HOSTNAME}

To check if the all workers in the cluster running is working correctly, run:

celery --app airflow.providers.celery.executors.celery_executor.app inspect ping

For more information, see: Management Command-line Utilities (inspect/control) and Workers Guide in the Celery documentation.