tests.system.openlineage.operator¶
Attributes¶
Classes¶
This operator is added for system testing purposes. |
Functions¶
|
|
|
|
|
|
|
|
|
Use this jinja method to access the environment variable named 'var'. |
|
|
|
|
|
|
|
Check if result is "equal" to expected value. |
Module Contents¶
- tests.system.openlineage.operator.env_var(var, default=None)[source]¶
Use this jinja method to access the environment variable named ‘var’.
If there is no such environment variable set, return the default. If the default is None, raise an exception for an undefined variable.
- tests.system.openlineage.operator.match(expected, result, env)[source]¶
Check if result is “equal” to expected value.
Omits keys not specified in expected value and resolves any jinja templates found.
Special sentinel values in expected dicts:
None(JSONnull) means the key must not be present in the result.{"$optional": "<reason>", ...}means the key is optional. When the key is present, the remaining template keys are validated. When the key is absent, validation passes silently and the reason is logged.{"$optional": "<reason>", "$value": <list_template>}is the list-template form of the optional sentinel. It behaves the same way, but also skips validation when the actual value is an empty list. Use this form when the template value is a list rather than a dict. Dict templates embed their keys directly alongside"$optional".
- class tests.system.openlineage.operator.OpenLineageTestOperator(event_templates=None, file_path=None, env=None, event_count_assertions=None, clear_variables=True, fail_fast=False, event_sort_fn=None, **kwargs)[source]¶
Bases:
airflow.providers.common.compat.sdk.BaseOperatorThis operator is added for system testing purposes.
It compares expected event templates set on initialization with ones emitted by OpenLineage integration and stored in Variables by VariableTransport.
- Note:
If clear_variables is True, only the Airflow Variables listed in event_templates (or derived from file_path) will be deleted - those that are supposed to be checked by the Operator. We won’t remove all Airflow Variables to avoid interfering with other instances of this Operator running in parallel on different DAGs. Running continuous system tests without clearing Variables may lead to leftover or growing Variables size. We recommend implementing a process to remove all Airflow Variables after all system tests have run to ensure a clean environment for each test run.
- Parameters:
event_templates (dict[str, dict | list] | None) – Dictionary where each key is the key used by VariableTransport, in the format
<DAG_ID>.<TASK_ID>.event.<EVENT_TYPE>. Each value may be one of the following: a single event template dict, matched against the last received event; a list of event template dicts, matched pairwise against all received events after sorting withevent_sort_fnwhen provided; or{"$absent": "<reason>"}, which asserts that no events exist for that key. The"$absent"reason is logged and included in the error if events are unexpectedly found.file_path (str | None) – alternatively, a path to a JSON file containing an array of event templates. Multiple events sharing the same
job.name+eventTypekey are accumulated into a list. An event with"$absent": "<reason>"asserts the variable must not exist.env (jinja2.Environment | None) – jinja environment used to render event templates.
event_count_assertions (dict[str, str | int] | None) – dict mapping a regex pattern (matched against variable keys) to a count expression. Supported expressions:
"N","==N",">=N","<=N". Applied to single-template keys only; list-template keys derive their count from the list length. Example:{".*\.event\.start": ">=2"}.clear_variables (bool) – if True, deletes only the checked variables after all checks run (or on failure). Default True.
fail_fast (bool) – if True, raise on the first failure. If False (default), collect all failures and raise a combined error at the end.
event_sort_fn (collections.abc.Callable[[dict], Any] | None) – optional callable
(event: dict) -> Anyused to sort actual events ascending before comparison. Useful for mapped tasks whose events arrive in non-deterministic order. When the template for a key is a list, sorting is required for pairwise comparison to be meaningful.
Special sentinel values in expected dicts:
None(JSONnull) means the key must not be present in the result.{"$optional": "<reason>", ...}means the key is optional. When the key is present, the remaining template keys are validated. When the key is absent, validation passes silently and the reason is logged.{"$optional": "<reason>", "$value": <list_template>}is the list-template form of the optional sentinel. It behaves the same way, but also skips validation when the actual value is an empty list. Use this form when the template value is a list rather than a dict. Dict templates embed their keys directly alongside"$optional".- Raises:
ValueError if the received events do not match with expected ones.