airflow.providers.snowflake.operators.snowpark_containers

Classes

SnowparkContainerJobOperator

Execute a job on Snowpark Container Services.

Module Contents

class airflow.providers.snowflake.operators.snowpark_containers.SnowparkContainerJobOperator(*, compute_pool, container_name, spec=None, spec_stage=None, spec_text=None, name=None, query_warehouse=None, replicas=1, external_access_integrations=None, wait_for_completion=True, drop_on_completion=True, poll_interval=10, snowflake_conn_id='snowflake_default', deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), timeout=24 * 60 * 60, database=None, schema=None, role=None, warehouse=None, **kwargs)[source]

Bases: airflow.providers.common.compat.standard.operators.BaseOperator

Execute a job on Snowpark Container Services.

Submits a container job to a compute pool via EXECUTE JOB SERVICE, optionally polls for completion, retrieves container logs, and drops the job service on success.

See also

Snowpark Container Services

Parameters:

  • compute_pool (str) – name of the compute pool to run the job on

  • container_name (str) – container name as defined in the service specification file, used for retrieving container logs

  • spec (str | None) – spec filename on the stage (e.g. 'spec.yaml'). Must be provided together with spec_stage

  • spec_stage (str | None) – stage where the spec file is stored (e.g. '@my_stage'). Must be provided together with spec

  • spec_text (str | None) – inline YAML spec text, as an alternative to spec/spec_stage. The text is wrapped in $$ delimiters automatically

  • name (str | None) – (Optional) job service name. If not provided, Snowflake auto-generates a name

  • query_warehouse (str | None) – (Optional) warehouse for SQL queries run inside the container. This is separate from the warehouse parameter used by the operator’s own SQL commands

  • replicas (int) – (Optional) number of job replicas to run. (default value: 1)

  • external_access_integrations (list[str] | None) – (Optional) Names of the external access integrations that allow your job to access external sites. Names are case-sensitive (default value: None)

  • wait_for_completion (bool) – poll until the job reaches a terminal state. When disabled, the job is submitted and the operator returns immediately. (default value: True)

  • drop_on_completion (bool) – drop the job service after the job finishes successfully or on a timeout. Failed jobs are not dropped, allowing inspection in Snowflake. (default value: True)

  • poll_interval (int) – the interval in seconds to poll the query status. (default value: 10)

  • snowflake_conn_id (str) – Reference to Snowflake connection id

  • deferrable (bool) – Run the operator in deferrable mode. Only effective when wait_for_completion is True. With wait_for_completion=False the operator submits the job and returns immediately without deferring. (default value: False)

  • timeout (int) – Maximum seconds to wait for the job to reach a terminal state. When it elapses the task fails. (default value: 86400)

  • database (str | None) – name of database (will overwrite database defined in connection)

  • schema (str | None) – name of schema (will overwrite schema defined in connection)

  • role (str | None) – name of role (will overwrite any role defined in connection’s extra JSON)

  • warehouse (str | None) – name of warehouse (will overwrite any warehouse defined in the connection’s extra JSON). Used for the operator’s own SQL commands, not for the container’s queries

template_fields: collections.abc.Sequence[str] = ('compute_pool', 'spec', 'spec_stage', 'container_name', 'spec_text', 'name', 'query_warehouse',...[source]
compute_pool[source]
container_name[source]
spec = None[source]
spec_stage = None[source]
spec_text = None[source]
name = None[source]
query_warehouse = None[source]
replicas = 1[source]
external_access_integrations = None[source]
wait_for_completion = True[source]
drop_on_completion = True[source]
poll_interval = 10[source]
snowflake_conn_id = 'snowflake_default'[source]
deferrable[source]
timeout = 86400[source]
database = None[source]
schema = None[source]
role = None[source]
warehouse = None[source]
job_name: str | None = None[source]
on_kill()[source]

Drop the running service on task kill.

execute(context)[source]

Submit and optionally wait for a Snowpark Container Services job.

execute_complete(context, event)[source]

Resume after the trigger fires.

Was this entry helpful?