Intuit, Inc.

Vigith Maurice

Amit Shriram Kalamkar

Edward Kihyen Lee

Juanlu Yu

Edward K Lee

Aspects of the present disclosure relate to watermarks and watermarking techniques for data streaming pipelines. Time stamp and offset timeline data is shared by computing instances along the pipeline to enable improved watermarking of the data stream through the pipeline. The improved watermarks enable better determination of completeness for the data stream and improve materialization of the results. The watermarking techniques can include periodically publishing watermark data by processing units of a vertex, fetching a merged watermark for a vertex by a vertex, and/or watching a data storage for the watermark data for events. Consensus algorithms can be used to maintain consensus among vertices for the watermark data.

Offset-based watermarks for data stream processing

A method including receiving events from different data sources for a service automatically executing in an enterprise system. A first event is enriched by providing the first event with first metadata that associates the first event with a first application used by the service. The first event is assigned to a time slice associated with the first application. A second event is enriched in a similar manner. A correlation graph of nodes and edges is built using the enriched events, with nodes representing the events and edges indicating relationships between the edges. A third event indicating a fault in the first application associated with the first node is received. The source of the error for the third event is identified using the second updated correlation graph and the time slice. The source of error is then mitigated.

Enriched high fidelity metrics

A method implements a streaming machine learning platform. The method includes processing event data with a preprocessing application to generate model input data and cardinality data in response to receiving the event data from a streaming application. The method further includes forwarding, using the cardinality data, the model input data to a machine learning model. The method further includes processing the model input data with the machine learning model to generate model output data stored to an output record. The method further includes processing the model output data from the output record with a post processing application to generate post data stored to a post record. The method further includes generating notification data using the post data of the post record and transmitting a notification of the notification data.

Streaming machine learning platform

A system including a data repository storing metrics describing operational behavior of software programs executing in an enterprise system. The system also includes an application programming interface (API) gateway configured to receive the metrics. The system also includes an ingestion layer configured to ingest the metrics to form the ingested metrics. The system also includes a tumbling window processor configured to process the ingested metrics and the events into heat maps, sort the heat maps into the time slices, and populate the time slices with the ingested metrics.

A method includes receiving first metrics describing first behaviors of applications having a fault, and second metrics describing second behaviors of an enterprise system in which the applications are executing. The first and second metrics are ingested. Ingesting includes assigning a first cardinality score to first metrics related to the fault. Ingesting also includes assigning a second, lower cardinality score to second metrics that are unrelated to the fault. Tumbling window processing is performed on the ingested metrics by sorting the first metrics and the second metrics into time slices. Each of the time slices includes a corresponding data structure for storing the data in the different time periods. Tumbling window processing is also performed on the ingested metrics by storing more data in the time slices for the first metrics having the first cardinality score relative to the second metrics having the second cardinality score.

Growing community of inventors

Mountain View, CA, United States of America

Vigith Maurice