airflow.providers.common.io.xcom.backend¶
Attributes¶
Classes¶
XCom backend that stores data in an object store or database depending on the size of the data. |
Module Contents¶
- class airflow.providers.common.io.xcom.backend.XComObjectStorageBackend[source]¶
Bases:
airflow.sdk.bases.xcom.BaseXComXCom backend that stores data in an object store or database depending on the size of the data.
If the value is larger than the configured threshold, it will be stored in an object store. Otherwise, it will be stored in the database. If it is stored in an object store, the path to the object in the store will be returned and saved in the database (by BaseXCom). Otherwise, the value itself will be returned and thus saved in the database.
- classmethod store_value(coordinate, key, value)[source]¶
Store an XCom value for
coordinateand return what its XCom row in the metadata database holds.Override this to keep values outside the metadata database, in any format. The return value must be JSON-serializable and must locate what was stored, since
load_value()andpurge()only receive the row. Every iteration of a task inside a loop is a separate task instance with the samedag_id,run_id,task_id,map_indexandkey, so the storage location must tell task instances apart.XComCoordinate.build_path()builds such a location. When a task instance, or its trigger, writes its own XCom,coordinate.ti_idis that task instance’s id, and a location unique per(coordinate.ti_id, key)is enough. When one task writes another’s XCom,coordinate.ti_idisNone, because only the server knows which task instance the coordinates resolve to, so the location must be unique to each write. A backend that does not usebuild_pathhas to handle both cases itself, or callsuper().The default stores the value in the metadata database, so an override can call
super()for the values it does not keep elsewhere.
- static serialize_value(value, *, key=None, task_id=None, dag_id=None, run_id=None, map_index=None)[source]¶
Serialize XCom value to JSON str.