SnowparkContainerJobOperator¶

Use the SnowparkContainerJobOperator to submit and monitor container jobs on Snowpark Container Services.

The operator wraps the Snowflake EXECUTE JOB SERVICE SQL command. It submits the job asynchronously, optionally polls until the job reaches a terminal state, retrieves container logs, and can drop the service on completion.

Prerequisite Tasks¶

To use this operator, you must do a few things:

Install the provider package via pip.
pip install 'apache-airflow-providers-snowflake'
Detailed information is available for Installation.
Setup a Snowflake Connection.

Create the necessary resources in your Snowflake account. See the Snowpark Container Services documentation.

Using the Operator¶

Use the snowflake_conn_id argument to specify the connection used. If not specified, snowflake_default will be used.

An example usage of the SnowparkContainerJobOperator is as follows:

tests/system/snowflake/example_snowpark_container_job.py[source]

run_job = SnowparkContainerJobOperator(
    task_id="run_job",
    compute_pool="my_compute_pool",
    container_name="main",
    spec="spec.yaml",
    spec_stage="@my_stage",
)

Note

Parameters that can be passed onto the operator will be given priority over the parameters already given in the Airflow connection metadata (such as schema, role, database and so forth).