Intuit, Inc.

Suman Ghosh

Saikiran Sri Thunuguntla

Vishal Reddy Baddam

Rajendra Tiwari

Mayur Madnani

Gururaj Uddihal

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives a partitioned incremental CDC changeset and combines the changes with the baseline match dataframe to produce a final changed baseline data table.

Batch materialization using bloom filters

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset with full row changes. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives full row changes from a partitioned incremental CDC changeset and combines the changes with the baseline unmatched dataframe to produce a final changed baseline data table.

Batch materialization for full row updates

Certain aspects of the present disclosure provide techniques for efficient data parity. Embodiments include receiving, by a first data consuming component, from a publication service, a plurality of database change records indicating changes to an underlying data source. Embodiments include updating, by the first data consuming component, a secondary data store based on the plurality of database change records. Embodiments include selecting a sample subset of the plurality of database change records based on one or more conditions. Embodiments include, for each respective database change record in the sample subset that has already been consumed by a second data consuming component that updated a primary data store based on one or more database change records: comparing the respective database change record to first data from the primary data store that corresponds to the respective database change record and making a parity determination based on the comparing.

Growing community of inventors

Hennur, India

Suman Ghosh