airtunnel.sensors package¶
Submodules¶
airtunnel.sensors.ingestion module¶
Module for Airtunnel’s ingestion sensors.
-
airtunnel.sensors.ingestion.
SourceFileIsReadySensor
(asset: airtunnel.data_asset.BaseDataAsset, no_of_required_static_pokes: int = 2, poke_interval: int = 30, timeout: int = 900, **kwargs)¶ Airtunnel’s SourceFileIsReadySensor – for a given Airtunnel data asset of type
ingested
, this sensor will leverage the declared input files glob pattern to sense for new input data.If files have been discovered, the sensor will capture their names and modification dates. Then it will check that for
no_of_required_static_pokes
times, file names and modification dates have not changed, before returning successfully.This guards against incomplete larger input files that are still being written into the landing area while this sensor finds them.
airtunnel.sensors.metadata module¶
Module for Airtunnel’s metadata sensors.
-
airtunnel.sensors.metadata.
AwaitAssetAncestorsUpdatedSensor
(asset: airtunnel.data_asset.BaseDataAsset, ignore_ancestors: Optional[Iterable[airtunnel.data_asset.BaseDataAsset]] = (), ancestors_refreshed_within: datetime.timedelta = None, include_upstream_levels: Iterable[int] = (0, ), poke_interval: int = 30, timeout: int = 900, **kwargs)¶ Airtunnel’s AwaitAssetAncestorsUpdatedSensor – using lineage for a given data asset, probes until all ancestors have been updated. (i.e. load status timestamp of all ancestors is “recent enough”)
-
airtunnel.sensors.metadata.
AwaitLoadStatusSensor
(asset: airtunnel.data_asset.BaseDataAsset, poke_interval: int = 30, timeout: int = 900, refreshed_within: datetime.timedelta = None, refreshed_after: datetime.datetime = None, **kwargs)¶ Airtunnel’s AwaitLoadStatusSensor – checks the data asset load status metadata for a given condition.
Module contents¶
Package for Airtunnel’s sensors.