airflow.providers.airbyte.operators.airbyte

Classes

AirbyteTriggerSyncOperator

Submits a job to an Airbyte server to run a integration process between your source and destination.

Module Contents

class airflow.providers.airbyte.operators.airbyte.AirbyteTriggerSyncOperator(connection_id, airbyte_conn_id='airbyte_default', asynchronous=False, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), api_version='v1', wait_seconds=3, timeout=3600, **kwargs)[source]

Bases: airflow.providers.common.compat.sdk.BaseOperator

Submits a job to an Airbyte server to run a integration process between your source and destination.

See also

For more information on how to use this operator, take a look at the guide: AirbyteTriggerSyncOperator

Parameters:

  • airbyte_conn_id (str) – Optional. The name of the Airflow connection to get connection information for Airbyte. Defaults to “airbyte_default”.

  • connection_id (str) – Required. The Airbyte ConnectionId UUID between a source and destination.

  • asynchronous (bool) – Optional. Flag to get job_id after submitting the job to the Airbyte API. This is useful for submitting long running jobs and waiting on them asynchronously using the AirbyteJobSensor. Defaults to False.

  • deferrable (bool) – Run operator in the deferrable mode.

  • api_version (str) – Optional. Airbyte API version. Defaults to “v1”.

  • wait_seconds (float) – Optional. Number of seconds between checks. Only used when asynchronous is False. Defaults to 3 seconds.

  • timeout (float) – Optional. The amount of time, in seconds, to wait for the request to complete. Only used when asynchronous is False. This limits how long the operator waits for the job to complete and does not imply job cancellation. Task-level timeouts should be enforced via execution_timeout. Defaults to 3600 seconds (or 1 hour).

  • execution_timeout – Maximum time allowed for the task to run. If exceeded, the Airbyte Job will be cancelled and the task will fail. When both execution_timeout and timeout are set, the earlier deadline takes precedence.

template_fields: collections.abc.Sequence[str] = ('connection_id',)[source]
ui_color = '#6C51FD'[source]
airbyte_conn_id = 'airbyte_default'[source]
connection_id[source]
timeout = 3600[source]
api_version = 'v1'[source]
wait_seconds = 3[source]
asynchronous = False[source]
deferrable[source]
execute(context)[source]

Create Airbyte Job and wait to finish.

execute_complete(context, event=None)[source]

Invoke this callback when the trigger fires; return immediately.

Relies on trigger to throw an exception, otherwise it assumes execution was successful.

on_kill()[source]

Cancel the job if task is cancelled.

Was this entry helpful?