airflow.providers.amazon.aws.operators.redshift_data¶

Classes¶

`ResumableJobMixin`	Airflow <3.3 stub, task_state_store unavailable, always submits fresh.
`RedshiftDataOperator`	Executes SQL Statements against an Amazon Redshift cluster using Redshift Data.

Module Contents¶

class airflow.providers.amazon.aws.operators.redshift_data.ResumableJobMixin(*, durable=True, **kwargs)[source]¶

Airflow <3.3 stub, task_state_store unavailable, always submits fresh.

external_id_key: str = 'redshift_statement_id'[source]¶

durable = True[source]¶

execute_resumable(context)[source]¶

class airflow.providers.amazon.aws.operators.redshift_data.RedshiftDataOperator(sql, database=None, cluster_identifier=None, db_user=None, parameters=None, secret_arn=None, statement_name=None, with_event=False, wait_for_completion=True, poll_interval=10, return_sql_result=False, workgroup_name=None, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), session_id=None, session_keep_alive_seconds=None, **kwargs)[source]¶

Bases: airflow.sdk.ResumableJobMixin, airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.redshift_data.RedshiftDataHook]

Executes SQL Statements against an Amazon Redshift cluster using Redshift Data.

… see also::: For more information on how to use this operator, take a look at the guide: Execute a statement on an Amazon Redshift cluster

Parameters:

database (str | None) – the name of the database
sql (str | list) – the SQL statement or list of SQL statement to run
cluster_identifier (str | None) – unique identifier of a cluster
db_user (str | None) – the database username
parameters (list | None) – the parameters for the SQL statement
secret_arn (str | None) – the name or ARN of the secret that enables db access
statement_name (str | None) – the name of the SQL statement
with_event (bool) – indicates whether to send an event to EventBridge
wait_for_completion (bool) – indicates whether to wait for a result, if True wait, if False don’t wait
poll_interval (int) – how often in seconds to check the query status
return_sql_result (bool) – if True will return the result of an SQL statement, if False (default) will return statement ID
workgroup_name (str | None) – name of the Redshift Serverless workgroup. Mutually exclusive with cluster_identifier. Specify this parameter to query Redshift Serverless. More info https://docs.aws.amazon.com/redshift/latest/mgmt/working-with-serverless.html
session_id (str | None) – the session identifier of the query
session_keep_alive_seconds (int | None) – duration in seconds to keep the session alive after the query finishes. The maximum time a session can keep alive is 24 hours
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
verify – Whether to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html
durable – When True (the default), the Redshift statement id is persisted to task state before polling begins. A worker crash on retry reconnects to the existing statement instead of resubmitting the SQL. Set to False to always submit fresh SQL on retry. Requires Airflow 3.3+; ignored silently on earlier versions.

aws_hook_class[source]¶

external_id_key = 'redshift_statement_id'[source]¶

template_fields[source]¶

template_ext = ('.sql',)[source]¶

template_fields_renderers[source]¶

database = None[source]¶

sql[source]¶

cluster_identifier = None[source]¶

workgroup_name = None[source]¶

db_user = None[source]¶

parameters = None[source]¶

secret_arn = None[source]¶

statement_name = None[source]¶

with_event = False[source]¶

wait_for_completion = True[source]¶

return_sql_result = False[source]¶

deferrable[source]¶

session_id = None[source]¶

session_keep_alive_seconds = None[source]¶

execute(context)[source]¶

Execute a statement against Amazon Redshift.

execute_complete(context, event=None)[source]¶

get_sql_results(statement_id, return_sql_result)[source]¶

Retrieve either the result of the SQL query, or the statement ID(s).

Parameters:

statement_id (str) – Statement ID of the running queries
return_sql_result (bool) – Boolean, true if results should be returned

on_kill()[source]¶

Cancel the submitted redshift query.

submit_job(context)[source]¶

Submit the statement for execution and return its statement id.

get_job_status(external_id, context)[source]¶

Query the raw statement status; a missing/expired statement degrades to NOT_FOUND.

is_job_active(status)[source]¶

Return True if the job is still running and can be reconnected to.

status is a raw string returned by the external system — not an Airflow enum. Its values are backend-specific (e.g. "RUNNING", "Pending", "ContainerCreating").

is_job_succeeded(status)[source]¶

Return True if the job completed successfully.

status is a raw string returned by the external system — not an Airflow enum. Its values are backend-specific (e.g. "FINISHED", "Succeeded").

poll_until_complete(external_id, context)[source]¶

Block until the job reaches a terminal state. Raise on failure.

get_job_result(external_id, context)[source]¶

Return the job result after completion. Return None if not applicable.