airtunnel.sensors package

Submodules

airtunnel.sensors.ingestion module

Module for Airtunnel’s ingestion sensors.

airtunnel.sensors.ingestion.SourceFileIsReadySensor(asset: airtunnel.data_asset.BaseDataAsset, no_of_required_static_pokes: int = 2, poke_interval: int = 30, timeout: int = 900, **kwargs)

Airtunnel’s SourceFileIsReadySensor – for a given Airtunnel data asset of type ingested, this sensor will leverage the declared input files glob pattern to sense for new input data.

If files have been discovered, the sensor will capture their names and modification dates. Then it will check that for no_of_required_static_pokes times, file names and modification dates have not changed, before returning successfully.

This guards against incomplete larger input files that are still being written into the landing area while this sensor finds them.

airtunnel.sensors.metadata module

Module for Airtunnel’s metadata sensors.

airtunnel.sensors.metadata.AwaitAssetAncestorsUpdatedSensor(asset: airtunnel.data_asset.BaseDataAsset, ignore_ancestors: Optional[Iterable[airtunnel.data_asset.BaseDataAsset]] = (), ancestors_refreshed_within: datetime.timedelta = None, include_upstream_levels: Iterable[int] = (0, ), poke_interval: int = 30, timeout: int = 900, **kwargs)

Airtunnel’s AwaitAssetAncestorsUpdatedSensor – using lineage for a given data asset, probes until all ancestors have been updated. (i.e. load status timestamp of all ancestors is “recent enough”)

airtunnel.sensors.metadata.AwaitLoadStatusSensor(asset: airtunnel.data_asset.BaseDataAsset, poke_interval: int = 30, timeout: int = 900, refreshed_within: datetime.timedelta = None, refreshed_after: datetime.datetime = None, **kwargs)

Airtunnel’s AwaitLoadStatusSensor – checks the data asset load status metadata for a given condition.

Module contents

Package for Airtunnel’s sensors.