Adaptive filtering and modeling via adaptive experimental designs to identify emerging data patterns from large volume, high dimensional, high velocity streaming data

Abstract

A system for identifying information in high dimensional, low latency streaming data having dynamically evolving data patterns. The system processes, continuously and in real-time, the streaming data. Processing includes filtering the data based on event data to identify diagnostic data points by comparing the event data with an experimental design matrix and performing a modeling operation using the identified diagnostic data points in order to identify efficiently any current and emerging patterns of relationships between at least one outcome variable and predictor variables. The at least one a-priori, pre-designed experimental design matrix is generated based on combinations of the predictor variables and at least one outcome variable. The experimental design matrix is also generated based on at least one of main effects, limitations, constraints, and interaction effects of the predictor variables and combinations.