airflow.providers.common.sql.operators.generic_transfer

Classes

GenericTransfer

Moves data from a connection to another.

Module Contents

class airflow.providers.common.sql.operators.generic_transfer.GenericTransfer(*, sql, destination_table, source_conn_id, source_hook_params=None, destination_conn_id, destination_hook_params=None, preoperator=None, insert_args=None, page_size=None, **kwargs)[source]

Bases: airflow.models.BaseOperator

Moves data from a connection to another.

Assuming that they both provide the required methods in their respective hooks. The source hook needs to expose a get_records method, and the destination a insert_rows method.

This is meant to be used on small-ish datasets that fit in memory.

Parameters:

  • sql (str) – SQL query to execute against the source database. (templated)

  • destination_table (str) – target table. (templated)

  • source_conn_id (str) – source connection. (templated)

  • source_hook_params (dict | None) – source hook parameters.

  • destination_conn_id (str) – destination connection. (templated)

  • destination_hook_params (dict | None) – destination hook parameters.

  • preoperator (str | list[str] | None) – sql statement or list of statements to be executed prior to loading the data. (templated)

  • insert_args (dict | None) – extra params for insert_rows method.

  • page_size (int | None) – number of records to be read in paginated mode (optional).

template_fields: collections.abc.Sequence[str] = ('source_conn_id', 'destination_conn_id', 'sql', 'destination_table', 'preoperator', 'insert_args')[source]
template_ext: collections.abc.Sequence[str] = ('.sql', '.hql')[source]
template_fields_renderers[source]
ui_color = '#b0f07c'[source]
sql[source]
destination_table[source]
source_conn_id[source]
source_hook_params = None[source]
destination_conn_id[source]
destination_hook_params = None[source]
preoperator = None[source]
insert_args[source]
page_size = None[source]
classmethod get_hook(conn_id, hook_params=None)[source]

Return DbApiHook for this connection id.

Parameters:

  • conn_id (str) – connection id

  • hook_params (dict | None) – hook parameters

Returns:

DbApiHook for this connection

Return type:

airflow.providers.common.sql.hooks.sql.DbApiHook

property source_hook: airflow.providers.common.sql.hooks.sql.DbApiHook[source]
property destination_hook: airflow.providers.common.sql.hooks.sql.DbApiHook[source]
get_paginated_sql(offset)[source]

Format the paginated SQL statement using the current format.

render_template_fields(context, jinja_env=None)[source]

Template all attributes listed in self.template_fields.

This mutates the attributes in-place and is irreversible.

Parameters:

  • context (airflow.sdk.definitions.context.Context) – Context dict with values to apply on content.

  • jinja_env (jinja2.Environment | None) – Jinja’s environment to use for rendering.

execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]

Was this entry helpful?