airflow.providers.amazon.aws.operators.athena_spark¶
Classes¶
Run an Apache Spark calculation in an Amazon Athena session. |
Module Contents¶
- class airflow.providers.amazon.aws.operators.athena_spark.AthenaSparkOperator(*, session_id, code_block, description=None, client_request_token=None, waiter_delay=30, waiter_max_attempts=120, log_query=True, aws_conn_id='aws_default', region_name=None, verify=None, botocore_config=None, **kwargs)[source]¶
Bases:
airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.athena.AthenaHook]Run an Apache Spark calculation in an Amazon Athena session.
Submits a calculation, such as PySpark code, via the Athena API, polls until the calculation reaches a terminal state, and returns execution metadata.
- Parameters:
session_id (str) – The Athena session ID in which to run the calculation. (templated)
code_block (str) – The calculation code, such as PySpark, to execute. (templated)
description (str | None) – Optional description of the calculation. Defaults to None.
client_request_token (str | None) – Optional idempotency token for the submission. Defaults to None.
waiter_delay (int) – Seconds to wait between status checks. Defaults to 30.
waiter_max_attempts (int) – Maximum number of polling attempts before timing out. Defaults to 120. To limit total task time, use execution_timeout on the task as well.
log_query (bool) – Whether to log submission details. Defaults to True.
aws_conn_id (str | None) – The Airflow connection used for AWS credentials. Defaults to
aws_default.region_name (str | None) – AWS region. If not set, default boto3 behavior is used. Defaults to None.
verify (bool | str | None) – Whether to verify SSL certificates. Defaults to None.
botocore_config (dict | None) – Optional botocore configuration dict. Defaults to None.
- template_fields: collections.abc.Sequence[str][source]¶
- template_ext: collections.abc.Sequence[str] = ('.py',)[source]¶