airflow.providers.huggingface.operators.datasets
Operators to load, transform, and publish Hugging Face datasets.
Module Contents
-
class airflow.providers.huggingface.operators.datasets.LoadDatasetOperator(*, path, output_uri, name=None, split=None, revision=None, cache_dir=None, storage_conn_id=None, download_mode='reuse_dataset_if_exists', huggingface_conn_id='huggingface_default', load_kwargs=None, durable=True, **kwargs)[source]
Bases: airflow.providers.common.compat.sdk.BaseOperator
Load a dataset revision and persist it as a Parquet artifact.
-
template_fields = ('path', 'name', 'split', 'revision', 'cache_dir', 'output_uri', 'storage_conn_id',...[source]
-
path[source]
-
output_uri[source]
-
name = None[source]
-
split = None[source]
-
revision = None[source]
-
cache_dir = None[source]
-
download_mode = 'reuse_dataset_if_exists'[source]
-
storage_conn_id = None[source]
-
huggingface_conn_id = 'huggingface_default'[source]
-
load_kwargs[source]
-
durable = True[source]
-
execute(context)[source]
Derive when creating an operator.
The main method to execute the task. Context is the same dictionary used
as when rendering jinja templates.
Refer to get_template_context for more context.
-
class airflow.providers.huggingface.operators.datasets.StreamDatasetOperator(*, path, repo_id, transform=None, predicate=None, name=None, split=None, revision=None, config=None, private=None, commit_message=None, transform_kwargs=None, push_kwargs=None, huggingface_conn_id='huggingface_default', durable=True, **kwargs)[source]
Bases: airflow.providers.common.compat.sdk.BaseOperator
Transform a streamed dataset lazily and publish it directly to the Hub.
-
template_fields = ('path', 'name', 'split', 'revision', 'config', 'commit_message', 'huggingface_conn_id')[source]
-
classmethod partial(**kwargs)[source]
-
path[source]
-
repo_id[source]
-
name = None[source]
-
split = None[source]
-
revision = None[source]
-
config = None[source]
-
private = None[source]
-
commit_message = None[source]
-
transform_kwargs[source]
-
push_kwargs[source]
-
huggingface_conn_id = 'huggingface_default'[source]
-
durable = True[source]
-
execute(context)[source]
Derive when creating an operator.
The main method to execute the task. Context is the same dictionary used
as when rendering jinja templates.
Refer to get_template_context for more context.
-
class airflow.providers.huggingface.operators.datasets.MapDatasetOperator(*, input_uri, output_uri, transform, transform_kwargs=None, map_kwargs=None, cache_dir=None, storage_conn_id=None, **kwargs)[source]
Bases: airflow.providers.common.compat.sdk.BaseOperator
Apply a Python callable and persist the transformed artifact.
-
template_fields = ('input_uri', 'output_uri', 'transform_kwargs', 'cache_dir', 'storage_conn_id')[source]
-
input_uri[source]
-
output_uri[source]
-
transform[source]
-
transform_kwargs = None[source]
-
map_kwargs[source]
-
cache_dir = None[source]
-
storage_conn_id = None[source]
-
execute(context)[source]
Derive when creating an operator.
The main method to execute the task. Context is the same dictionary used
as when rendering jinja templates.
Refer to get_template_context for more context.
-
class airflow.providers.huggingface.operators.datasets.PushToHubOperator(*, input_uri, repo_id, config=None, private=None, commit_message=None, huggingface_conn_id='huggingface_default', storage_conn_id=None, push_kwargs=None, durable=True, **kwargs)[source]
Bases: airflow.providers.common.compat.sdk.BaseOperator
Synchronously publish an artifact and emit its successful asset event.
-
template_fields = ('input_uri', 'config', 'commit_message', 'huggingface_conn_id', 'storage_conn_id')[source]
-
classmethod partial(**kwargs)[source]
-
input_uri[source]
-
repo_id[source]
-
config = None[source]
-
private = None[source]
-
commit_message = None[source]
-
huggingface_conn_id = 'huggingface_default'[source]
-
push_kwargs[source]
-
storage_conn_id = None[source]
-
durable = True[source]
-
execute(context)[source]
Derive when creating an operator.
The main method to execute the task. Context is the same dictionary used
as when rendering jinja templates.
Refer to get_template_context for more context.