American Express Travel Related Services Company, Inc.

Rahul Agarwal

Harish R Naik

Sachin Arya

Sastry Vsm Durvasula

Sandeep Bose

Satish Thenthiruperai Narayanan

Ajay Paul Singh Manesh

Neha Singh

The system may validate a data source having a structured format and a grammar that includes tags. The system may identify a tag in the grammar. The system may parse the data source to extract attributes and/or values associated with the tags in response to successful validation. The system may also write the attributes and/or values to an output file separated by a preselected delimiter. A configuration file may identify the grammar, the preselected delimiter, and/or the data source. The data source may be in an XML format or a JSON format. The system may generate execution ready code in response to the validating the data source and the grammar. The output file may be a load ready file for ingestion into a big data storage format. The tag may include a parent tag and a sub tag corresponding to a hierarchy in the data source.

Systems and methods for parsing and ingesting data in big data environments

A system receives a source and a metadata layer that describes the source. The source may comprise source records with fields containing source data, and the metadata layer may include metadata comprising at least one of a field data type, a field data length, a field description, or a record length. The processor may further validate the metadata layer against the source and write results to a log. The processor may further be configured for transforming the source records into transformed records for a load ready file. The processor may further balance a number of records in the source against a number of transformed records in the load ready file to generate a transformation failure rate.

Systems and methods for data validation and processing using metadata

Growing community of inventors

Noida, India

Rahul Agarwal