Capital One Services, LLC

Abhijit Chitnis

Behdad Forghani

Naga Venkata Sriram Vadakattu

Aditya Relangi

Aniketh Bothra

Naga Chintala

Gurucharan Manadavadi Prakash

Madan Gumudavelly

Sarvani Maddina

Parvesh Kumar

Anurag Jain

Janardhan Deepak Prabhakara

Rahul Surendra Nath

Mayur Jagtap

Mark Assousa

Methods and systems described herein may include processing a change data capture stream in parallel and optimizing the write operations contained in the change data capture stream such that the write operations may be written to a change table in parallel. Processing the change data capture stream may include decoding the transaction data contained in the change data capture stream in parallel. After the transaction data in the change data capture stream has been decoded, the transaction data may be analyzed to identify write operations that write to the same row. Write operations that write to the same row may be merged into a single write operation. The merging of write operations may be performed until each write operation writes to a different row. The write operations may then be written to the change table in parallel.

Parallel stream processing of change data capture

The exemplary embodiments provide real-time data capture and processing which improves data processing performance and speed and facilitate passing of the processed data to various analytical sources, while maintaining superior data quality checks, particularly with respect to data elements associated with multiple data types. The proposed system and process can be used to continuously consume and listen to multiple events while mapping the events to appropriate schemas provided in a separate schema stream. The schema stream is provided once and cached to minimize bandwidth consumed by the transaction stream. The schema information is then further enriched with information from a metadata registry. The event data may then be compressed and aligned in memory tables based on the enriched schema. Once events are decoded and sorted into memory tables in accordance to the identified schema, each memory table can be processed in parallel.

System, method and computer-accessible medium for capturing data changes

Disclosed herein are various embodiments for a data transfer and resource management system. An embodiment operates by retrieving both data and a schema from a table. A first offset corresponding to the retrieved data indicating a retrieval time of the retrieved data and a number of records is identified. It is determined that the retrieved data corresponds to the schema. The data is stored on a messaging platform and made available to be pulled by one or more end users prior to a transfer to one or more cloud servers. A second offset corresponding to a number of records of the data retrieved by the one or more end users from the messaging platform is determined. It is validate that the second offset corresponds to the first offset. The records and the schema retrieved from the intermediary messaging platform are transferred to the one or more cloud servers.

Data transfer and resource management system

Disclosed herein are system, method, and computer program product embodiments for a data transfer and resource management system. An embodiment operates by retrieving both data and a schema from a table. A first offset corresponding to the retrieved data indicating a retrieval time of the retrieved data and a number of records is identified. It is determined that the retrieved data corresponds to the schema. The data is stored on a messaging platform and made available to one or more end users prior to a transfer to one or more cloud servers. A second offset corresponding to a number of records of the data retrieved by the one or more end users from the messaging platform is determined. It is validate that the second offset corresponds to the first offset. The records and the schema retrieved from the intermediary messaging platform are transferred to the one or more cloud servers.

Growing community of inventors

Frisco, TX, United States of America

Abhijit Chitnis