Capital One Services, LLC

Thomas J O'Connor

Jose Moreno

Jon Stofer

William Ye

Victor P Viner

Daniel Liu

Alec Sokol

Saianirudh Kantabathina

Scott Mandrell

Richard Hightower

Navaneet Galagali

David Woods

Serhiy Kulyk

Mary Baum

Michael Qiu

Sathi Chowdhury

Christopher Cellucci

Lisa Y Lehman

Selwyn Lehmann

Patrick Ryan

Forrest Thomas

Disclosed herein are system, method, and computer program product embodiments for performing in-flight transformations of data streams. Complex event processing (CEP) operating on event records from various data streams can operate by storing the streamed event record data in a database, and then querying the data to perform a data transformation. However, in order to improve the performance of the data transformations and streamline CEP, a serverless architecture is introduced that can perform data transformations directly on streams, using attributes of the streamed record data defined in a schema. The resulting transformed data can then be provided by the serverless architecture to the CEP for direct access of the data most needed by the CEP.

In-flight transformations over a data stream

In certain embodiments, machine learning and lineage data may be used to manage data. In some embodiments, a computing system may use lineage data to identify two datasets that may be related. The computing system may determine that a user has access to a derivative dataset but does not have access to an original dataset that was used to create the derivative dataset. In response, the computing system may use a machine learning model to generate a similarity score indicating a level of similarity between the original dataset and the derivative dataset. If the similarity score satisfies a threshold score, the computing system may modify access rights of the user so that the user is unable to access a portion of the data in the derivative dataset.

Systems and methods for using machine learning to manage data

Foot massager

Growing community of inventors

McLean, VA, United States of America

Thomas J O'Connor