Multimodal data reduction agent for high density data in iiot applications

Abstract

Data reduction services are implemented on one or more nodes of an IIoT data pipeline to intelligently determine an appropriate data reduction strategy based on characteristics of the incoming data. In one or more embodiments, data reduction components on the pipeline node or on an edge device define different data filtering rules or algorithms that are selectively applied to streaming time-series data based on a probability distribution of the data. The data pipeline node performs real-time distribution analysis on the streaming data to determine whether the data has a unimodal distribution, a multimodal distribution, or no mode, and selects one of the data filtering rules based on this determined probability distribution. In this way, the data is intelligently reduced in a manner that retains critical information within the reduced data set while achieving a high level of data reduction.