SnowparkContainerJobOperator¶

Use the SnowparkContainerJobOperator to submit and monitor container jobs on Snowpark Container Services.

The operator wraps the Snowflake EXECUTE JOB SERVICE SQL command. It submits the job asynchronously, optionally polls until the job reaches a terminal state, retrieves container logs, and can drop the service on completion.

You can also run this operator in deferrable mode by setting deferrable param to True. This will ensure that the task is deferred from the Airflow worker slot and polling for the job status happens on the trigger.

Note

In deferrable mode, clearing or marking a deferred task as failed drops the job service only on Airflow 3.3 or later. On earlier versions the service is left running and billing.

Prerequisite Tasks¶

To use this operator, you must do a few things:

Install the provider package via pip.
pip install 'apache-airflow-providers-snowflake'
Detailed information is available for Installation.
Setup a Snowflake Connection.

Create the necessary resources in your Snowflake account. See the Snowpark Container Services documentation.

Using the Operator¶

Use the snowflake_conn_id argument to specify the connection used. If not specified, snowflake_default will be used.

An example usage of the SnowparkContainerJobOperator is as follows:

tests/system/snowflake/example_snowpark_container_job.py[source]

run_job = SnowparkContainerJobOperator(
    task_id="run_job",
    compute_pool="my_compute_pool",
    container_name="main",
    spec="spec.yaml",
    spec_stage="@my_stage",
)

Note

Parameters that can be passed onto the operator will be given priority over the parameters already given in the Airflow connection metadata (such as schema, role, database and so forth).