Airflow Summit 2026 is coming August 31 - September 2 in Austin, TX. Register now to secure your spot!

AWS Glue

AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. AWS Glue provides all the capabilities needed for data integration so that you can start analyzing your data and putting it to use in minutes instead of months.

Prerequisite Tasks

To use these operators, you must do a few things:

Generic Parameters

aws_conn_id

Reference to Amazon Web Services Connection ID. If this parameter is set to None then the default boto3 behaviour is used without a connection lookup. Otherwise use the credentials stored in the Connection. Default: aws_default

region_name

AWS Region Name. If this parameter is set to None or omitted then region_name from AWS Connection Extra Parameter will be used. Otherwise use the specified value instead of the connection value. Default: None

verify

Whether or not to verify SSL certificates.

  • False - Do not validate SSL certificates.

  • path/to/cert/bundle.pem - A filename of the CA cert bundle to use. You can specify this argument if you want to use a different CA cert bundle than the one used by botocore.

If this parameter is set to None or is omitted then verify from AWS Connection Extra Parameter will be used. Otherwise use the specified value instead of the connection value. Default: None

botocore_config

The provided dictionary is used to construct a botocore.config.Config. This configuration can be used to configure Avoid Throttling exceptions, timeouts, etc.

Example, for more detail about parameters please have a look botocore.config.Config
{
    "signature_version": "unsigned",
    "s3": {
        "us_east_1_regional_endpoint": True,
    },
    "retries": {
      "mode": "standard",
      "max_attempts": 10,
    },
    "connect_timeout": 300,
    "read_timeout": 300,
    "tcp_keepalive": True,
}

If this parameter is set to None or omitted then config_kwargs from AWS Connection Extra Parameter will be used. Otherwise use the specified value instead of the connection value. Default: None

Note

Specifying an empty dictionary, {}, will overwrite the connection configuration for botocore.config.Config

Operators

Create an AWS Glue crawler

AWS Glue Crawlers allow you to easily extract data from various data sources. To create a crawler, use GlueCrawlerCreateOperator.

tests/system/amazon/aws/example_glue.py[source]

create_crawler = GlueCrawlerCreateOperator(
    task_id="create_crawler",
    config=glue_crawler_config,
)

Note

The AWS IAM role included in the config needs access to the source data location (e.g. s3:PutObject access if data is stored in Amazon S3) as well as the AWSGlueServiceRole policy. See the References section below for a link to more details.

Update an AWS Glue crawler

To update the configuration of an existing crawler, use GlueCrawlerUpdateOperator.

tests/system/amazon/aws/example_glue.py[source]

update_crawler = GlueCrawlerUpdateOperator(
    task_id="update_crawler",
    config=updated_glue_crawler_config,
)

Run an AWS Glue crawler

To run an existing crawler and wait for it to complete, use GlueCrawlerRunOperator.

tests/system/amazon/aws/example_glue.py[source]

run_crawler = GlueCrawlerRunOperator(
    task_id="run_crawler",
    crawler_name=glue_crawler_name,
)

The operator waits for completion by default. Set deferrable=True to perform the wait without occupying a worker slot.

Delete an AWS Glue crawler

To delete an existing crawler, use GlueCrawlerDeleteOperator.

tests/system/amazon/aws/example_glue.py[source]

delete_crawler = GlueCrawlerDeleteOperator(
    task_id="delete_crawler",
    crawler_name=glue_crawler_name,
    trigger_rule=TriggerRule.ALL_DONE,
)

Legacy AWS Glue crawler operator

Warning

GlueCrawlerOperator is deprecated. Existing Dags can continue using it during the deprecation period, but new Dags should use the operation-specific operators above.

The legacy operator creates or updates a crawler and then runs it. Existing Dags can continue using the same configuration while migrating each operation to the dedicated operators:

crawl_s3 = GlueCrawlerOperator(
    task_id="crawl_s3",
    config=glue_crawler_config,
)

Submit an AWS Glue job

To submit a new AWS Glue job you can use GlueJobOperator.

tests/system/amazon/aws/example_glue.py[source]

submit_glue_job = GlueJobOperator(
    task_id="submit_glue_job",
    job_name=glue_job_name,
    script_location=f"s3://{bucket_name}/etl_script.py",
    s3_bucket=bucket_name,
    iam_role_name=role_name,
    create_job_kwargs={"GlueVersion": "3.0", "NumberOfWorkers": 2, "WorkerType": "G.1X"},
)

Note

The same AWS IAM role used for the crawler can be used here as well, but it will need policies to provide access to the output location for result data.

A Glue job run that ends in STOPPED is treated as a failure, not a success – on every attempt, first or retry, regardless of durable. Glue’s API has no way to tell a run cancelled manually (for example, in the AWS console) apart from one this operator’s own on_kill() stopped, which happens whenever stop_job_run_on_kill=True and the task is killed – on SIGTERM, on execution_timeout, or when the task is cleared while running. The task fails and a normal retry resubmits, rather than a self-inflicted stop being silently reported as a false success. Setting durable=False does not change this.

Durable execution

GlueJobOperator submits a job run and then polls it to completion on the worker. By default the operator runs in a durable mode that makes this crash-safe: the Glue job run id is persisted to task state store before polling begins, so if the worker crashes or is preempted and the task is retried, the operator reconnects to the run that is already executing in Glue instead of starting a new one.

This matters more for Glue because a Glue job’s concurrent_run_limit defaults to 1, so submitting a second run while the first is still active does not create a harmless duplicate, it fails outright with ConcurrentRunsExceededException and the task keeps retrying against a run it can never see. Durable execution turns that retry into a normal reconnect.

On retry the operator checks the prior run’s state:

  • if it is still starting, running, waiting for capacity, or being stopped, the operator reconnects and continues polling

  • if it already succeeded, the operator returns immediately without resubmitting

  • if it failed terminally, or its id has expired and is no longer found, the operator submits the job fresh

  • if it already stopped, the operator submits the job fresh

If the stored state is already STOPPED, the operator submits fresh rather than reconnecting to it. If a reconnect finds the run still stopping and it settles into STOPPED while polling, the operator raises instead of returning a result – see above, this applies regardless of durable.

This protection also applies when wait_for_completion=False – even though that task attempt never polls at all, a retry after a successful submission still reconnects rather than resubmitting, since the run id is persisted immediately after submission regardless of whether the task waits for it to finish.

Durable execution requires Airflow 3.3 or newer for the task state store lookup above. Below 3.3, durable has no effect: setting it explicitly only emits a warning, and its value is ignored either way. The deprecated resume_glue_job_on_retry parameter is the only way to opt into crash recovery there, and it still works via an older mechanism: the operator checks XCom for a cached run id first, then falls back to scanning the job’s run history for a run tagged with this task instance’s identity, and reconnects if it finds one that is still active. The XCom check only ever succeeds on Airflow 2.x – every Airflow 3 release clears task XComs before each non-deferral attempt, so on Airflow 3.0-3.2 every retry pays the full scan.

That older mechanism only activates below 3.3 when resume_glue_job_on_retry=True is set explicitly – durable=True does not turn it on there. Upgrading the provider alone, with no DAG change, does not turn it on either: below 3.3, behavior is unchanged from before this feature existed unless resume_glue_job_on_retry is set. On Airflow 3.3+, durable defaults to True as described above, since the task state store makes it cheap.

Like the persisted state itself, the stored run id isn’t deleted automatically, that only happens when someone runs airflow state-store clean or airflow db clean (which also targets the task_state_store table). If a task’s retry_delay is longer than [state_store] default_retention_days (30 days by default) and cleanup runs in between, the run id won’t be there for the next retry, and the operator falls back to the XCom/scan mechanism above rather than reconnecting via task state store. Avoid running cleanup on a schedule shorter than your longest retry_delay.

Clearing a task is treated the same as a retry, which matters specifically for a task whose job already succeeded: clearing does not delete the stored run id, so the next attempt reads it back and returns immediately without submitting anything to Glue. See Resumable Tasks for why, and for the [state_store] clear_on_success setting that restores “clearing always resubmits.”

To opt out and always start a fresh run on retry, set durable=False:

glue_job = GlueJobOperator(
    task_id="glue_job",
    job_name="my_glue_job",
    script_location="s3://glue-examples/glue-scripts/sample_aws_glue_job.py",
    durable=False,
)

The task state store lookup above is only used on the synchronous path – when deferrable=True is set, the Triggerer already tracks the run across the wait, so a run id is never persisted there. durable still has an effect on retry, though: a retry of a deferrable task would otherwise resubmit, and with concurrent_run_limit=1 that fails with ConcurrentRunsExceededException against the run it can’t see. To avoid that, durable=True (or, below Airflow 3.3, resume_glue_job_on_retry=True) tags the job’s arguments with this task instance’s identity on every attempt and, on retry, scans the job’s run history for that tag before submitting – the same mechanism used as a fallback on the synchronous path.

durable supersedes the deprecated resume_glue_job_on_retry parameter on Airflow 3.3+, where passing resume_glue_job_on_retry still works and maps its value onto durable. Below 3.3, resume_glue_job_on_retry remains the only working option, since durable is a no-op there. Either way, passing it emits an AirflowProviderDeprecationWarning, since the parameter will be removed once this provider’s minimum supported Airflow version reaches 3.3.

Create an AWS Glue Data Quality

AWS Glue Data Quality allows you to measure and monitor the quality of your data so that you can make good business decisions. To create a new AWS Glue Data Quality ruleset or update an existing one you can use GlueDataQualityOperator.

tests/system/amazon/aws/example_glue_data_quality.py[source]

create_rule_set = GlueDataQualityOperator(
    task_id="create_rule_set",
    name=rule_set_name,
    ruleset=RULE_SET,
    data_quality_ruleset_kwargs={
        "TargetTable": {
            "TableName": athena_table,
            "DatabaseName": athena_database,
        }
    },
)

Start a AWS Glue Data Quality Evaluation Run

To start a AWS Glue Data Quality ruleset evaluation run you can use GlueDataQualityRuleSetEvaluationRunOperator.

tests/system/amazon/aws/example_glue_data_quality.py[source]

start_evaluation_run = GlueDataQualityRuleSetEvaluationRunOperator(
    task_id="start_evaluation_run",
    datasource={
        "GlueTable": {
            "TableName": athena_table,
            "DatabaseName": athena_database,
        }
    },
    role=test_context[ROLE_ARN_KEY],
    rule_set_names=[rule_set_name],
)

Start a AWS Glue Data Quality Recommendation Run

To start a AWS Glue Data Quality rule recommendation run you can use GlueDataQualityRuleRecommendationRunOperator.

tests/system/amazon/aws/example_glue_data_quality_with_recommendation.py[source]

recommendation_run = GlueDataQualityRuleRecommendationRunOperator(
    task_id="recommendation_run",
    datasource={
        "GlueTable": {
            "TableName": athena_table,
            "DatabaseName": athena_database,
        }
    },
    role=test_context[ROLE_ARN_KEY],
    recommendation_run_kwargs={"CreatedRulesetName": rule_set_name},
)

Sensors

Wait on an AWS Glue crawler state

To wait on the state of an AWS Glue crawler execution until it reaches a terminal state you can use GlueCrawlerSensor.

tests/system/amazon/aws/example_glue.py[source]

wait_for_crawl = GlueCrawlerSensor(
    task_id="wait_for_crawl",
    crawler_name=glue_crawler_name,
)

Wait on an AWS Glue job state

To wait on the state of an AWS Glue Job until it reaches a terminal state you can use GlueJobSensor

tests/system/amazon/aws/example_glue.py[source]

wait_for_job = GlueJobSensor(
    task_id="wait_for_job",
    job_name=glue_job_name,
    # Job ID extracted from previous Glue Job Operator task
    run_id=submit_glue_job.output,
    verbose=True,  # prints glue job logs in airflow logs
)

Wait on an AWS Glue Data Quality Evaluation Run

To wait on the state of an AWS Glue Data Quality RuleSet Evaluation Run until it reaches a terminal state you can use GlueDataQualityRuleSetEvaluationRunSensor

tests/system/amazon/aws/example_glue_data_quality.py[source]

await_evaluation_run_sensor = GlueDataQualityRuleSetEvaluationRunSensor(
    task_id="await_evaluation_run_sensor",
    evaluation_run_id=start_evaluation_run.output,
)

Wait on an AWS Glue Data Quality Recommendation Run

To wait on the state of an AWS Glue Data Quality recommendation run until it reaches a terminal state you can use GlueDataQualityRuleRecommendationRunSensor

tests/system/amazon/aws/example_glue_data_quality_with_recommendation.py[source]

await_recommendation_run_sensor = GlueDataQualityRuleRecommendationRunSensor(
    task_id="await_recommendation_run_sensor",
    recommendation_run_id=recommendation_run.output,
)

Wait on an AWS Glue Catalog Partition

To wait for a partition to show up in AWS Glue Catalog until it reaches a terminal state you can use GlueCatalogPartitionSensor

tests/system/amazon/aws/example_glue.py[source]

wait_for_catalog_partition = GlueCatalogPartitionSensor(
    task_id="wait_for_catalog_partition",
    table_name="input",
    database_name=glue_db_name,
    expression="category='mixed'",
)

Reference

Was this entry helpful?