airflow.providers.apache.hive.sensors.hive_partition

Module Contents

Classes

HivePartitionSensor

Waits for a partition to show up in Hive.
class airflow.providers.apache.hive.sensors.hive_partition.HivePartitionSensor(*, table, partition="ds='{{ ds }}'", metastore_conn_id='metastore_default', schema='default', poke_interval=60 * 3, **kwargs)[source]

Bases: airflow.sensors.base.BaseSensorOperator

Waits for a partition to show up in Hive.

Note: Because partition supports general logical operators, it can be inefficient. Consider using NamedHivePartitionSensor instead if you don’t need the full flexibility of HivePartitionSensor.

Parameters

  • table (str) – The name of the table to wait for, supports the dot notation (my_database.my_table)

  • partition (str | None) – The partition clause to wait for. This is passed as is to the metastore Thrift client get_partitions_by_filter method, and apparently supports SQL like notation as in ds='2015-01-01' AND type='value' and comparison operators as in "ds>=2015-01-01"

  • metastore_conn_id (str) – reference to the :ref: metastore thrift service connection id <howto/connection:hive_metastore>

template_fields: collections.abc.Sequence[str] = ('schema', 'table', 'partition')[source]
ui_color = '#C5CAE9'[source]
poke(context)[source]

Override when deriving this class.

Was this entry helpful?