American Express Travel Related Services Company, Inc.

Ravi K Kaushik

Ravi Arasan

Sandeep Bose

Matthew Kent Meyer

Jeremy David Seideman

Gurusamy Ramasamy

Debasish Das

Ambika Aggarwal

Marimuthu Ponnambalam

Kunal Chandrashekhar Joshi

Himanshu Prabhakar

Atul R Dalmia

Pratiksha M Jogani

Vineet Agrwal

Kandarp Srivastava

Systems and methods of the present disclosure may read source data corresponding to a source variable and apply a transformation to the source variable to generate an output variable. Accordingly, one such method comprises receiving a request to access an output variable by applying one or more data transformations to input data; before applying an actual data transformation to the input data, identifying that a match exists with a database of prior data transformations that have been applied to source variables to generate derived variables; identifying that a source variable in the matching one or more data transformations is flagged as sensitive data; and processing the request to access the output variable subject to an access restriction in response to the identification of the source variable that is flagged as sensitive data.

Lineage data for data records

Systems and methods of the present disclosure may read source data corresponding to a source variable and apply a transformation to the source variable to generate an output variable. Accordingly, one such method comprises receiving a request to generate an output variable by applying one or more data transformations to a source variable; before applying an actual data transformation to the source variable, checking a database of prior data transformations that have been applied to source variables to generate existing output variables for a match with the requested one or more data transformations to the source variable; determining that a match exists in the database, wherein the match is associated with an existing output variable; and/or responding to the request by providing the existing output variable associated with the match that exists in the database of prior data transformations.

The process includes receiving a data set comprising a plurality of rows and a plurality of columns, and applying a first rule based decisioning to the data set to generate a first layer of metadata that comprises at least one of a key, a type indicator, a categorical indicator, and/or a continuous indicator. The first layer of metadata may be descriptive of the data set. The processor may further apply a second rule based decisioning to the first layer to generate a second layer that includes at least one of the key, the type indicator, the categorical indicator, or the continuous indicator. The second layer may be descriptive of the first layer. The process may also include generating an output file from at least one of the first layer or the second layer.

Rule based decisioning on metadata layers

A system may read source data corresponding to a source variable and apply a transformation to the source variable to generate an output variable. The transformation may include logic, and the output variable may be configured for ingestion into a big data storage format. The system may record lineage data of the output variable that comprises the transformation and/or the source variable. The system may also receive a request to generate a requested output variable. The requested output variable may be generated from a second transformation that is the same as the first transformation. The system may thus match the first transformation to the second transformation using the lineage data. In response to matching the first transformation to the second transformation, the system may deny the request. The original output variable may be returned in response to the matching the first transformation to the second transformation.

A system may store a plurality of first records comprising a first variable in a first data storage format. The system may also store a plurality of second records comprising a second variable in a second data storage format. The first data storage format may have a different data structure than the second data storage format. The system may then generate a catalog including a first location of the first variable and a second location of the second variable. The first location identifies the first data storage format, which is where the first variable is stored. The second location identifies the second data storage format, which is where the second variable is stored. The system may then receive a request to access at least one of the first variable or the second variable and access the first variable and/or the second variable in response to the request.

Catalog with location of variables for data

Integrated big data interface for multiple storage types

System and method for automatically capturing and recording lineage data for big data records

System and method for recursive metadata layers on big data sets

Growing community of inventors

New York, NY, United States of America

Ravi K Kaushik