Lineage

Note

Lineage support is very experimental and subject to change.

Airflow provides a powerful feature for tracking data lineage not only between tasks but also from hooks used within those tasks. This functionality helps you understand how data flows throughout your Airflow pipelines.

A global instance of HookLineageCollector serves as the central hub for collecting lineage information. Hooks can send details about assets they interact with to this collector. The collector then uses this data to construct AIP-60 compliant Assets, a standard format for describing assets. Hooks can also send arbitrary non-asset related data to this collector as shown in the example below.

from airflow.sdk.lineage import get_hook_lineage_collector


class CustomHook(BaseHook):
    def run(self):
        # run actual code
        collector = get_hook_lineage_collector()
        collector.add_input_asset(self, asset_kwargs={"scheme": "file", "path": "/tmp/in"})
        collector.add_output_asset(self, asset_kwargs={"scheme": "file", "path": "/tmp/out"})
        collector.add_extra(self, key="external_system_job_id", value="some_id_123")

Lineage data collected by the HookLineageCollector can be accessed using an instance of HookLineageReader, which is registered in an Airflow plugin.

from airflow.sdk.lineage import HookLineageReader
from airflow.plugins_manager import AirflowPlugin


class CustomHookLineageReader(HookLineageReader):
    def get_inputs(self):
        return self.lineage_collector.collected_assets.inputs


class HookLineageCollectionPlugin(AirflowPlugin):
    name = "HookLineageCollectionPlugin"
    hook_lineage_readers = [CustomHookLineageReader]

If no HookLineageReader is registered within Airflow, a default NoOpCollector is used instead. This collector does not create AIP-60 compliant assets or collect lineage information.

Limiting collected lineage

A collector keeps at most max_assets_per_collector input assets and the same number of output assets, counted separately, along with max_extras_per_collector extra metadata entries. Entries beyond a limit are dropped, so a hook called in a long loop stops contributing lineage once its limit is reached. Setting a limit to 0 disables that kind of collection entirely.