airflow.providers.amazon.aws.hooks.duckdb

Attributes

CredentialStrategy

DEFAULT_AWS_CONN_ID

Classes

AwsDuckDBHook

Interact with DuckDB using AWS credentials brokered by Airflow.

Module Contents

airflow.providers.amazon.aws.hooks.duckdb.CredentialStrategy[source]
airflow.providers.amazon.aws.hooks.duckdb.DEFAULT_AWS_CONN_ID = 'aws_default'[source]
class airflow.providers.amazon.aws.hooks.duckdb.AwsDuckDBHook(*args, aws_conn_id=NOTSET, region_name=None, credential_strategy=None, credential_chain=None, s3_endpoint_url=None, secret_name=None, **kwargs)[source]

Bases: airflow.providers.duckdb.hooks.duckdb.DuckDBHook

Interact with DuckDB using AWS credentials brokered by Airflow.

Extends DuckDBHook with an S3 secret built from an Airflow AWS connection, so read_parquet('s3://...') and COPY ... TO 's3://...' work with no credential wiring in the Dag.

Two credential strategies are supported:

credential_chain (the default)

Issues CREATE SECRET (TYPE s3, PROVIDER credential_chain), which resolves credentials through the AWS SDK inside DuckDB’s aws extension. Nothing secret is written into SQL text, and because DuckDB holds the resolution itself the SDK refreshes expiring credentials, so a query that outlives a set of temporary credentials does not fail partway through.

config

Resolves credentials through AwsBaseHook and writes them into the secret explicitly. Also works with a container-assigned role, since boto3 resolves that on the Airflow side, but the credentials are frozen at connect time and become part of the SQL statement. Needed when the connection carries static keys the AWS SDK running inside DuckDB cannot see, or when talking to an S3-compatible endpoint with its own credentials. Prefer credential_chain where it works.

Either strategy is better than none: DuckDB’s httpfs extension has its own HTTP client which reads credentials from the standard AWS_* environment variables but does not implement the ECS container credential provider, so an ambient task role is invisible to it and S3 access fails with an opaque HTTP 403. A session with no secret at all also ignores the Airflow connection entirely, including its region and endpoint.

This hook needs the httpfs and aws extensions and installs them if they are missing, which is where it diverges from DuckDBHook. Using this hook says the task runs against AWS, so those two extensions should just work. Set autoinstall_extensions=False to forbid the download, in which case the extensions have to be present already, either in an extension_directory or baked into the image.

Parameters:

  • aws_conn_id (str | None | airflow.providers.amazon.version_compat.ArgNotSet) – the AWS connection used to reach S3. Defaults to aws_default. Set to None to use no Airflow connection and let the AWS SDK resolve credentials from the environment instead.

  • region_name (str | None) – region for the S3 secret. Defaults to the region from aws_conn_id.

  • credential_strategy (CredentialStrategy | None) – credential_chain, config, or none to create no secret at all (for a DuckDB database that never touches S3).

  • credential_chain (str | None) – providers DuckDB’s credential chain consults, for example "env;config;sts;instance". Only used with the credential_chain strategy; DuckDB’s own default order applies when it is not set.

  • s3_endpoint_url (str | None) – override the S3 endpoint, for an S3-compatible service or a VPC endpoint. Defaults to endpoint_url from the AWS connection extra.

  • secret_name (str | None) – name of the DuckDB secret to create.

As with DuckDBHook, every parameter may also be set in the DuckDB connection extra, and an explicit argument wins. aws_conn_id resolves on whether the key is present, so {"aws_conn_id": null} selects the ambient AWS environment.

required_extensions = ('httpfs', 'aws')[source]
conn_type = 'duckdb_aws'[source]
hook_name = 'DuckDB on AWS'[source]
default_conn_name = 'duckdb_aws_default'[source]
classmethod get_ui_field_behaviour()[source]

Return custom UI field behaviour for the DuckDB on AWS connection.

property aws_conn_id: str | None[source]

Return the AWS connection to take credentials from, or None for the ambient environment.

Resolved by key presence rather than through resolve_parameter, because None is a valid choice here and not the absence of one: {"aws_conn_id": null} in the extra asks for no Airflow connection at all.

property aws_hook: airflow.providers.amazon.aws.hooks.base_aws.AwsBaseHook[source]

Return the AWS hook credentials and region are resolved through.

property autoinstall_extensions: bool[source]

Default to installing missing extensions, unlike the generic hook.

Reaching for the AWS hook is a statement that the task runs against AWS, so httpfs and aws should just work. The generic hook cannot assume that and leaves downloads off.

property region_name: str | None[source]
property credential_strategy: CredentialStrategy[source]
property credential_chain: str | None[source]
property secret_name: str[source]
get_region_name()[source]

Return the region for the S3 secret.

get_s3_endpoint_url()[source]

Return the S3 endpoint override, if any.

configure_secrets(conn)[source]

Create the DuckDB S3 secret for this connection.

get_openlineage_database_info(connection)[source]

Return no lineage metadata; a DuckDB database is task-local and has no stable namespace.

Was this entry helpful?