• Home
• Controlling what OpenLineage emits

↑↓ Navigate ⏎ Select Esc Close

Controlling what OpenLineage emits

This page is the one place to look when you want to control which OpenLineage events are emitted, how much metadata they contain, and who is allowed to override what.

Kill switch: disabled (turn OpenLineage off entirely)

If you want to turn the OpenLineage integration off completely — no plugin loaded, no listener registered, no events generated, no overhead — without uninstalling the provider, set:

[openlineage]
disabled = true

or the equivalent environment variable:

AIRFLOW__OPENLINEAGE__DISABLED=true

Note

Changing this variable may require Airflow restart, as the plugins (including OpenLineage) are loaded at startup.

This is the all-or-nothing kill switch — useful for incident response, for testing without OL side-effects, or to keep the provider installed but inactive. When disabled = true no emission_policy rule is evaluated and no authoring override has any effect: the OL plugin is not loaded.

For anything less than all-or-nothing — disabling specific DAGs, tasks, operator classes, or specific facets; controlling source-code capture or hook-lineage extraction; locking admin policy against per-Dag author overrides — use emission_policy instead, documented below.

Two layers of control

When OL is enabled, there are two layers of control, sitting on different sides of the wire:

1. The Airflow OpenLineage provider (this provider) controls what the producer generates in the first place — whether to run the extractor, whether to include source code, whether to emit task or DAG-run events at all, whether to inject the full AirflowRunFacet, and so on. This is the emission_policy Airflow configuration, documented below.

2. The OpenLineage Python client controls what happens to events after they are built — dropping events by name pattern, applying transport-level rules. See OpenLineage Python client – filters.

The provider runs first (skipping work) and the client runs second (reshaping what survives).

Unified emission_policy configuration

The emission_policy option (under the [openlineage] section) accepts a JSON array of rule objects. Each rule has this shape:

{
  "scope":      {<scope keys>},          # required; use {} for "global"
  "match_mode": "exact" | "regex",       # optional; default "exact"
  "controls":   {<control flags>},       # required; must be non-empty
  "locked":     true | false             # optional; default false
}

Top-level keys outside scope / match_mode / controls / locked are rejected and the rule is skipped with a WARNING. The same applies to unknown keys inside scope or controls. This catches typos like "dgg_id" immediately at config load.

Scope keys (all optional inside scope):

• operator — fully-qualified operator class name; applies to task events for that operator type.

• dag_id — applies to task and/or DAG-level events for the named DAG (use emit_task_events / emit_dag_events in controls to target one type selectively).

• task_id — only valid alongside dag_id; targets a specific task.

• hook: fully-qualified class name (module.ClassName) of the object that reported a hook-collected dataset, usually a hook such as airflow.providers.amazon.aws.hooks.s3.S3Hook. It can be added to any of the other scopes and narrows the rule to the datasets that hook reports. See Excluding datasets (exclude_datasets).

• (empty ``scope: {}``) — global default; applies to every task and DAG event.

scope must be one of: global (empty), operator-only, dag-only, or dag+task, each optionally with hook. task_id without dag_id, or operator combined with dag_id / task_id, is rejected with a WARNING.

``match_mode`` (top-level): "exact" (default) or "regex". When "regex", every value inside scope (dag_id, task_id, operator, hook) is treated as a re.fullmatch pattern.

Control flag keys (all optional inside controls; the dict must be non-empty):

Flag	Default	Description
emit	true	Shorthand: disable all OpenLineage events in scope (both task and DAG-level).
emit_task_events	true	Disable task-level events only; takes precedence over emit for task events within the same rule.
emit_dag_events	true	Disable DAG-run-level events only; takes precedence over emit for DAG events within the same rule.
extract_operator_metadata	true	Whether to run operator-specific extractor-based metadata collection (inputs/outputs). Task events only.
include_source_code	true	Whether to include source code in Python/Bash operator events. Task events only. No effect when ``extract_operator_metadata`` is ``false`` — the entire extraction pipeline is skipped.
hook_lineage	true	Whether to use HookLineageCollector as a fallback when the extractor finds no inputs/outputs. Task events only. No effect when ``extract_operator_metadata`` is ``false`` — the entire extraction pipeline (including hook lineage) is skipped.
include_full_task_info	false	Whether to include the full serialized operator state in AirflowRunFacet. When false, only a curated subset of task attributes is sent. When true, all serializable task parameters are included, which may significantly increase event size. Task events only.
exclude_datasets	[]	List of regex patterns; datasets whose <namespace>/<name> matches any of them are dropped. Task events only. See Excluding datasets (exclude_datasets).

Valid scope and control combinations. A rule that uses a control its scope does not accept is skipped with a WARNING that names the offending keys.

Control	Global ({})	operator	dag_id	dag_id + task_id	Any of these + hook
emit, emit_task_events	yes	yes	yes	yes	no
emit_dag_events	yes	no	yes	no	no
extract_operator_metadata, include_source_code, include_full_task_info	yes	yes	yes	yes	no
hook_lineage	yes	yes	yes	yes	yes, keeps or drops the hook’s datasets; not with exclude_datasets
exclude_datasets	yes	yes	yes	yes	yes; not with hook_lineage

locked (top-level, default false) is an admin-only floor lock: when true, the control fields carried by this rule’s controls dict cannot be overridden by per-Dag / per-task authoring flags (see Per-Dag / per-task authoring overrides (extend_global_openlineage_emission_policy)). The rule still participates in normal tier resolution against other conf rules — locking only blocks authoring overrides.

emit_task_events / emit_dag_events take precedence over emit within the same rule. For example, {"scope": {"dag_id": "X"}, "controls": {"emit": false, "emit_task_events": true}} suppresses DAG-level events for X while leaving task events enabled.

Priority for task events (most specific tier wins; within a tier, the last matching rule wins):

1. dag_id + task_id

2. dag_id

3. operator

4. Global (empty scope)

5. Built-in defaults

Priority for DAG-level events:

1. dag_id rule (using emit_dag_events or emit)

2. Global (empty scope)

3. Built-in defaults

Note

Contradictory rules within the same tier. When two rules in the same priority tier set the same flag to different values, the last rule in the list wins. A WARNING is logged to help you spot accidental conflicts. For example, [{"scope": {}, "controls": {"emit": false}}, {"scope": {}, "controls": {"emit": true}}] (both global) results in emit=true with a warning. Ordering rules intentionally is cleaner than relying on last-wins.

[openlineage]
emission_policy = [
  {"scope": {"operator": "airflow.providers.http.operators.http.HttpOperator"}, "controls": {"emit": false}},
  {"scope": {"operator": "airflow.providers.standard.operators.python.PythonOperator"}, "controls": {"include_source_code": false}},
  {"scope": {"dag_id": "expensive_dag"}, "controls": {"extract_operator_metadata": false}},
  {"scope": {"dag_id": "expensive_dag", "task_id": "send_report"}, "controls": {"extract_operator_metadata": true}},
  {"scope": {"dag_id": "my_dag", "task_id": "sensitive_task"}, "controls": {"hook_lineage": false}},
  {"scope": {"dag_id": "reporting_dag"}, "controls": {"emit_dag_events": false}},
  {"scope": {"dag_id": "full_control_dag"}, "controls": {"emit": false}},
  {"scope": {"dag_id": "full_control_dag", "task_id": "critical_task"}, "controls": {"emit": true}},
  {"scope": {"dag_id": "^staging_.*"}, "match_mode": "regex", "controls": {"emit": false}},
  {"scope": {"dag_id": "audit_dag"}, "controls": {"include_full_task_info": true}},
  {"scope": {}, "controls": {"include_source_code": false}}
]

AIRFLOW__OPENLINEAGE__EMISSION_POLICY environment variable is an equivalent.

In the example above:

• HttpOperator task events are suppressed entirely for all DAGs.

• PythonOperator tasks emit events but without source code (all DAGs).

• Tasks in expensive_dag skip dataset extraction — except send_report, which overrides this with a more-specific task-scoped rule and extracts normally.

• sensitive_task in my_dag runs its extractor normally but skips the HookLineageCollector fallback.

• DAG-level events for reporting_dag are suppressed; task events are still emitted.

• Both task events and DAG-level events for full_control_dag are suppressed — except critical_task, which re-enables task event emission via a task-scoped rule.

• All events for any DAG whose ID starts with staging_ are suppressed (regex match).

• Full task info is included in all task events for audit_dag.

• Source code is disabled globally for all operators (PythonOperator was already covered above, but the global rule applies to BashOperator and any other operator with source code).

Excluding datasets (exclude_datasets)

exclude_datasets drops individual datasets while keeping the rest of a task’s lineage, for example the per-chunk objects of a multipart upload that hook lineage reports one by one.

• Patterns are always matched with re.fullmatch, independent of match_mode, against <namespace>/<name> of the OpenLineage dataset: s3://my-bucket/staging/part_1 for a dataset with namespace s3://my-bucket and name staging/part_1. Names that already start with /, such as file paths, get no extra /: file://host1/staging/part_1 for namespace file://host1 and name /staging/part_1, and file/tmp/part_1 for a local file with namespace file. Airflow assets are matched after translation to OpenLineage, so assets that cannot be translated are never reported, whatever the patterns.

• They apply to every source of task lineage: extractor results, hook-collected assets, manually declared inlets / outlets, runtime outlet events, the per-query events emitted from SQL hook lineage, and the emit_dataset_lineage / emit_query_lineage APIs.

• Only datasets are dropped, never events. A SQL hook lineage query event, or an emit_query_lineage event, is still emitted when every dataset of the query is excluded, because it also carries the query text and id. emit_dataset_lineage skips its event in that case, since the event exists only to carry the datasets.

• Excluding datasets never changes which source is used. If an extractor returns datasets that are all excluded, hook lineage is not consulted as a fallback; if hook lineage collected assets that are all excluded, manually declared inlets / outlets are not used either. To report your own assets instead of the ones a hook collects, for example one asset for an upload prefix instead of one per uploaded part, set hook_lineage: false for the task and declare the asset in outlets.

• A rule whose scope has a hook key only applies to datasets reported by that hook, and only to hook lineage and SQL hook lineage (the other sources have no reporting hook). Such a rule may only carry exclude_datasets or hook_lineage; with a hook scope, hook_lineage: false drops every dataset the hook reports and hook_lineage: true keeps them all. A hook-scoped hook_lineage does not affect whether hook lineage is collected.

The list is resolved per dataset, and the most specific matching rule that sets it replaces broader ones instead of merging with them, so a narrower rule can shorten a broader exclusion or clear it with []. For a dataset reported by a hook, each tier is preceded by its hook variant:

1. dag_id + task_id + hook

2. dag_id + task_id

3. dag_id + hook

4. dag_id

5. operator + hook

6. operator

7. hook

8. Global (empty scope)

Every replacement is logged at INFO level, naming the winning rule and the broader rules it replaced. locked keeps its usual meaning: a locked rule blocks authoring overrides of exclude_datasets for the datasets it applies to, but a more specific conf rule still replaces its list.

[openlineage]
emission_policy = [
  {"scope": {"hook": ".*\\.S3Hook"}, "match_mode": "regex", "controls": {"exclude_datasets": ["s3://.*/_temporary/.*"]}},
  {"scope": {"dag_id": "chunked_dag", "task_id": "upload", "hook": "airflow.providers.amazon.aws.hooks.s3.S3Hook"}, "controls": {"exclude_datasets": ["s3://my-bucket/staging/part_.*"]}},
  {"scope": {"dag_id": "noisy_dag", "hook": "airflow.providers.google.cloud.hooks.gcs.GCSHook"}, "controls": {"hook_lineage": false}}
]

In the example above:

• _temporary objects reported by S3Hook are dropped in every Dag.

• upload in chunked_dag drops the staging parts its S3Hook writes. Its list replaces the first rule’s, so _temporary objects are kept for that task.

• Every dataset GCSHook reports in noisy_dag is dropped; other hooks in that Dag are unaffected.

Note

Exclusion happens after the collector cap. Hook-collected assets are filtered when OpenLineage reads the hook lineage collector, after [lineage] max_assets_per_collector has limited what the collector kept. Excluding assets reduces what gets reported, but it does not make room for assets the cap already discarded. The same applies to SQL hook lineage queries and [lineage] max_extras_per_collector.

Note

Only the event inputs / outputs lists are filtered. Facets can still refer to an excluded dataset: columnLineage on the remaining datasets, and the task’s inlets, outlets and runtime outlet assets in the airflow run facet.

Audit logging: whenever a control flag is non-default (suppressed or enabled), an INFO-level log is emitted identifying the field, the event context, and the exact rule that caused the change. This provides a clear audit trail for operational troubleshooting.

Per-Dag / per-task authoring overrides (extend_global_openlineage_emission_policy)

Dag authors can override most emission_policy flags directly in Dag code, without touching Airflow configuration, via extend_global_openlineage_emission_policy(). This sits above emission_policy conf rules in the resolution stack — the value provided in code wins unless the matching conf rule is marked with "locked": true.

from airflow.providers.openlineage.api.emission_policy import extend_global_openlineage_emission_policy

with DAG("my_dag", ...) as dag:
    extract = PythonOperator(task_id="extract", ...)
    sensitive = PythonOperator(task_id="sensitive", ...)

# Disable all events for one task
extend_global_openlineage_emission_policy(sensitive, emit=False)

# Disable source-code capture for every task in the Dag, re-enable for one
extend_global_openlineage_emission_policy(dag, include_source_code=False)
extend_global_openlineage_emission_policy(extract, include_source_code=True)

# Suppress DAG-run events but keep task events
extend_global_openlineage_emission_policy(dag, emit_dag_events=False)

# Drop the staging objects a task writes, keep the rest of its lineage
extend_global_openlineage_emission_policy(extract, exclude_datasets=["s3://my-bucket/staging/.*"])

Key semantics:

• Resolution priority: authoring flags > unlocked conf rules > built-in defaults.

• DAG-or-task only — no global authoring scope. extend_global_openlineage_emission_policy only accepts a single DAG, operator, or XComArg. There is no API for “apply to every DAG in the deployment.” Deployment-wide changes are an admin concern and belong in emission_policy with an empty "scope": {} (see the unified configuration above). Passing any other object type raises TypeError.

• Locked conf rules win: a rule with "locked": true blocks authoring overrides for the field(s) it carries. The override is silently ignored and an INFO-level log records the lock hit. This is a floor lock — any matching conf rule that locks a field is honored, even if another (more specific) conf rule wins the value resolution.

• DAG-level calls only propagate to existing tasks — flags set on a Dag are pushed to the tasks present on the Dag at the moment the call is made. Tasks defined later (e.g. inside with DAG(...) as dag: before the operators are declared) will not inherit them. Call extend_global_openlineage_emission_policy(dag, ...) after defining the tasks, or set flags per task.

• ``exclude_datasets`` has no hook dimension: an authored list applies to every dataset of the task and replaces the conf lists, including hook-scoped ones, except where a locked rule applies.

• No unset API: pass an explicit boolean to override; passing None means “not provided” and leaves any previously-stored value intact.

• Legacy options cannot be locked. "locked": true is an emission_policy feature. If you rely only on legacy options (see Legacy configuration options (deprecated) below), Dag authors can override them via extend_global_openlineage_emission_policy. To enforce a real lock, move the constraint into emission_policy and add "locked": true.

See the docstring of extend_global_openlineage_emission_policy() for the full list of supported flags and examples.

Legacy configuration options (deprecated)

Deprecated since version 2.19.0: The legacy options disabled_for_operators, disable_source_code, include_full_task_info, and selective_enable are superseded by emission_policy. They continue to work for now but will be removed in a future version — migrate by translating them into the equivalent rules shown below.

Any active legacy options are automatically translated into equivalent emission_policy rules. The translated rules are prepended before your explicit rules, so your explicit rules win within each priority tier (last-wins within a tier). A DeprecationWarning is issued listing every translated option — migrate your configuration to emission_policy exclusively to silence it.

The translation mapping is:

Legacy option	Translated to
disabled_for_operators = some.Operator	{"scope": {"operator": "some.Operator"}, "controls": {"emit": false}} (operator-scoped, one per entry)
disable_source_code = True	{"scope": {}, "controls": {"include_source_code": false}} (global rule)
include_full_task_info = True	{"scope": {}, "controls": {"include_full_task_info": true}} (global rule)
selective_enable = True	{"scope": {}, "controls": {"emit": false}} (global baseline) + {"scope": {"dag_id": "...", "task_id": "..."}, "controls": {"emit": true}} injected at runtime for each task that called enable_lineage()

Full translation example. Given this configuration:

[openlineage]
emission_policy        = [{"scope": {"dag_id": "my_dag"}, "controls": {"include_source_code": true}}]
disabled_for_operators = airflow.providers.standard.operators.python.PythonOperator
disable_source_code    = True
include_full_task_info = True
selective_enable       = True

The effective rule set evaluated at runtime becomes (legacy rules first, your rules last):

Tier: operator
  {"scope": {"operator": "airflow.providers.standard.operators.python.PythonOperator"},
   "controls": {"emit": false}}
  ↳ from disabled_for_operators

Tier: global
  {"scope": {}, "controls": {"include_source_code": false}}
  ↳ from disable_source_code

  {"scope": {}, "controls": {"include_full_task_info": true}}
  ↳ from include_full_task_info

  {"scope": {}, "controls": {"emit": false}}
  ↳ from selective_enable (global baseline — all tasks off by default)

Tier: task  (injected at runtime for each opted-in task)
  {"scope": {"dag_id": "my_dag", "task_id": "opted_in_task"}, "controls": {"emit": true}}
  ↳ from selective_enable + enable_lineage(task)

Your explicit emission_policy rules (appended last, win within tier):
Tier: dag
  {"scope": {"dag_id": "my_dag"}, "controls": {"include_source_code": true}}
  ↳ from emission_policy

Result for PythonOperator task in my_dag (task opted in via enable_lineage):

• emit — task-tier opt-in rule sets true, overriding the global false baseline.

• include_source_code — your explicit dag-tier rule overrides the global include_source_code=false from the translated legacy rule, because dag tier beats global tier.

• include_full_task_info — global true from the translated legacy rule.

Note

Because disabled_for_operators translates to an operator-tier rule, overriding it from emission_policy requires an operator-tier rule with a higher list position (i.e., appearing later in the same emission_policy list). A global {"scope": {}, "controls": {"emit": true}} rule will not override it — operator tier beats global tier. To re-enable a specific operator that is listed in disabled_for_operators, add an explicit operator rule to emission_policy:

[openlineage]
emission_policy        = [{"scope": {"operator": "my.pkg.MyOperator"}, "controls": {"emit": true}}]
disabled_for_operators = my.pkg.MyOperator

The translated legacy rule {"scope": {"operator": "my.pkg.MyOperator"}, "controls": {"emit": false}} is prepended, but the explicit {"scope": {"operator": "my.pkg.MyOperator"}, "controls": {"emit": true}} follows it in the same operator tier — last-wins, so the operator is re-enabled.

Previous Next

Was this entry helpful?