Intuit, Inc.

Vishal Reddy Baddam

Saikiran Sri Thunuguntla

Suman Ghosh

Pradeep Srinivas Krishna

Rajendra Tiwari

Thatchinamoorthy Kallipatti Arumugam

Gururaj Uddihal

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives a partitioned incremental CDC changeset and combines the changes with the baseline match dataframe to produce a final changed baseline data table.

Batch materialization using bloom filters

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset with full row changes. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives full row changes from a partitioned incremental CDC changeset and combines the changes with the baseline unmatched dataframe to produce a final changed baseline data table.

Batch materialization for full row updates

Systems and methods for intelligently repairing data are disclosed. An example method is performed by one or more processors of a data quality management (DQM) system and includes receiving a transmission over a communications network from a computing device associated with the DQM system, the transmission including an indication that source data stored in a source database was ingested and stored as target data in a target database at a time of ingestion, comparing, using an advanced DQM algorithm, the target data with the source data, the advanced DQM algorithm including generating a first set of parity results based on changes occurring before the time of ingestion, generating a second set of parity results based on changes occurring after the time of ingestion, and generating differential results based on the first and the second set of parity results, and selectively repairing ones of the changes based on the differential results.

Growing community of inventors

Bengaluru, India

Vishal Reddy Baddam