airflow.providers.amazon.aws.operators.glue_crawler¶

Classes¶

`GlueCrawlerCreateOperator`	Create an AWS Glue crawler.
`GlueCrawlerUpdateOperator`	Update an existing AWS Glue crawler.
`GlueCrawlerRunOperator`	Run an existing AWS Glue crawler.
`GlueCrawlerDeleteOperator`	Delete an AWS Glue crawler.
`GlueCrawlerOperator`	Create, update and run an AWS Glue crawler.

Module Contents¶

class airflow.providers.amazon.aws.operators.glue_crawler.GlueCrawlerCreateOperator(*, config, **kwargs)[source]¶

Bases: _GlueCrawlerBaseOperator

Create an AWS Glue crawler.

See also

For more information on how to use this operator, take a look at the guide: Create an AWS Glue crawler

Parameters:

config (dict[str, Any]) – Configurations for the AWS Glue crawler.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

template_fields: collections.abc.Sequence[str][source]¶

config[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.amazon.aws.operators.glue_crawler.GlueCrawlerUpdateOperator(*, config, **kwargs)[source]¶

Bases: _GlueCrawlerBaseOperator

Update an existing AWS Glue crawler.

See also

For more information on how to use this operator, take a look at the guide: Update an AWS Glue crawler

Parameters:

config (dict[str, Any]) – Configurations for the AWS Glue crawler.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

template_fields: collections.abc.Sequence[str][source]¶

config[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.amazon.aws.operators.glue_crawler.GlueCrawlerRunOperator(*, crawler_name, poll_interval=5, wait_for_completion=True, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), fail_on_already_running=True, **kwargs)[source]¶

Bases: _GlueCrawlerBaseOperator

Run an existing AWS Glue crawler.

AWS Glue Crawler is a serverless service that manages a catalog of metadata tables that contain the inferred schema, format and data types of data stores within the AWS cloud.

See also

For more information on how to use this operator, take a look at the guide: Run an AWS Glue crawler

Parameters:

crawler_name (str) – Name of the AWS Glue crawler.
poll_interval (int) – Time (in seconds) to wait between two consecutive calls to check crawler status
wait_for_completion (bool) – Whether to wait for crawl execution completion. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the crawl to complete. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
fail_on_already_running (bool) – If True (default), the operator will raise an exception when start_crawler() encounters a CrawlerRunningException (i.e., the crawler is already running). If False, the operator continues with the existing run and waits for it if wait_for_completion is True. Setting this to False is useful for handling retry-induced race conditions where boto3 retries trigger a second start_crawler() call after a network timeout on the first successful call. (default: True)
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

template_fields: collections.abc.Sequence[str][source]¶

poll_interval = 5[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

fail_on_already_running = True[source]¶

crawler_name[source]¶

execute(context)[source]¶

Run an AWS Glue crawler from Airflow.

Returns:: The name of the current Glue crawler.
Return type:: str

execute_complete(context, event=None)[source]¶

class airflow.providers.amazon.aws.operators.glue_crawler.GlueCrawlerDeleteOperator(*, crawler_name, **kwargs)[source]¶

Bases: _GlueCrawlerBaseOperator

Delete an AWS Glue crawler.

See also

For more information on how to use this operator, take a look at the guide: Delete an AWS Glue crawler

Parameters:

crawler_name (str) – Name of the AWS Glue crawler.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

template_fields: collections.abc.Sequence[str][source]¶

crawler_name[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.amazon.aws.operators.glue_crawler.GlueCrawlerOperator(config, poll_interval=5, wait_for_completion=True, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), fail_on_already_running=True, **kwargs)[source]¶

Bases: GlueCrawlerRunOperator

Create, update and run an AWS Glue crawler.

Deprecated since version Use: GlueCrawlerCreateOperator, GlueCrawlerUpdateOperator, and GlueCrawlerRunOperator instead.

Parameters:

config (dict[str, Any]) – Configurations for the AWS Glue crawler.
poll_interval (int) – Time (in seconds) to wait between two consecutive calls to check crawler status.
wait_for_completion (bool) – Whether to wait for crawl execution completion. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the crawl to complete. This implies waiting for completion. (default: False)
fail_on_already_running (bool) – If True (default), raise an exception if the crawler is already running. If False, wait for the existing run to complete.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether or not to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

template_fields: collections.abc.Sequence[str][source]¶

config[source]¶

execute(context)[source]¶

Run an AWS Glue crawler from Airflow.

Returns:: The name of the current Glue crawler.
Return type:: str

execute_complete(context, event=None)[source]¶