Intuit, Inc.

The Mathworks, Inc.

Saikiran Sri Thunuguntla

Vijaykumar Hiremath

Sreenivasulu Nallapati

Vishal Reddy Baddam

Suman Ghosh

Anishkumar Ss

Pradeep Srinivas Krishna

John Edward Ciolfi

Sanjai Singh

Raman Gupta

Yogesh Dalal

Thatchinamoorthy Kallipatti Arumugam

Keshav Bhashyam

Rajendra Tiwari

Gururaj Uddihal

Sidharth Kumar

Vasanth Kumar Jagadeesh

Sirsha Chatterjee

Senthil Kumar Ls

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives a partitioned incremental CDC changeset and combines the changes with the baseline match dataframe to produce a final changed baseline data table.

Batch materialization using bloom filters

Systems and methods are described for batch materialization of an incremental change data capture (CDC) changeset with full row changes. The primary keys are extracted from the incremental CDC changeset and an indication of the extracted primary keys are broadcast to a plurality of executors. The primary keys may be added to Bloom filter or a plurality of Bloom filters that are broadcast to the executors. Each executor filters a baseline data table based on the extracted primary keys to generate a baseline match dataframe with all primary keys matching the extracted primary keys, and a baseline unmatched dataframe with all primary keys not matching the extracted primary keys. Each executor receives full row changes from a partitioned incremental CDC changeset and combines the changes with the baseline unmatched dataframe to produce a final changed baseline data table.

Batch materialization for full row updates

Systems and methods for intelligently repairing data are disclosed. An example method is performed by one or more processors of a data quality management (DQM) system and includes receiving a transmission over a communications network from a computing device associated with the DQM system, the transmission including an indication that source data stored in a source database was ingested and stored as target data in a target database at a time of ingestion, comparing, using an advanced DQM algorithm, the target data with the source data, the advanced DQM algorithm including generating a first set of parity results based on changes occurring before the time of ingestion, generating a second set of parity results based on changes occurring after the time of ingestion, and generating differential results based on the first and the second set of parity results, and selectively repairing ones of the changes based on the differential results.

Intelligent data repair for moving source

An access graph is constructed based on access policy data from user accounts, data lake buckets, and/or access policy statements from any other location. Access logs are analyzed to determine actual access to the data tables. For a given user role, an initial set of data tables that are actually accessed is generated forming the baseline of data tables for which access privileges are to be maintained. User roles that are similar to the given user role are identified and additional data tables accessed by the similar user roles are added to the initial set of data tables to generate a final set of data tables. Access privileges to the final set of data tables are maintained for the given user role, while access privileges to the remaining data tables may be revoked.

Fine grained access control in a data lake using least privilege access

Systems and methods for updating a data catalog in real-time with changes to metadata are disclosed. An example method may be performed by one or more processors of a system and include detecting changes to metadata in a metadata store, generating an event queue indicating a plurality of change events associated with system objects based on the changes, each of the plurality of change events indicating a number of event attributes and a system object associated with the change event, identifying recent events among the plurality of change events, identifying relevant events among the recent events, extracting unique identifiers from the relevant events, identifying priority objects among the system objects, obtaining, from the metadata store, current metadata for each of the priority objects, and updating information associated with the priority objects in a data catalog, the updating occurring in real-time with the changes to the metadata in the metadata store.

Real-time crawling

Systems and methods for automated techniques that generate queryable database table ownership attribution information in real-time. In addition to generating ownership attribution information, system and methods provide a novel framework for creating bi-partite graphs and generating insightful graph data.

Automated database ownership attribution

Incoming data requests from the perspective of the data lake, are gathered and analyzed to determine the usage of the data. Using the perspective of the data lake avoids the technical challenge of analyzing data usage by different computation points, which are at different locations, perform hard-to-track different operations, and are often reachable only through complicated access protocols. Another technical challenge of mapping between an object path and a table path is solved by generating object path datasets and table path datasets at different levels of abstractions. A comparison is performed, iteratively, from a lower level of granularity and the granularity is increased in the progressive steps. Matches from iterations are unionized to generate a final matching data. Observability metrics are generated using final matching data and are used to perform downstream operations such as controlling data table access, moving data tables to cold storage, decommissioning unused pipelines, etc.

Generating observability metrics for data lake usage based on data layer activity logs

Systems and methods for generating augmented search results are disclosed. An example method may be performed by one or more processors of a ranking system and include receiving a search query, generating, for each respective data asset of a plurality of data assets, a query relevancy score indicating a likelihood that the respective data asset is relevant to the search query, retrieving, for each of a number of potentially relevant data assets, a utility score indicating a likelihood that the potentially relevant data asset is useful, generating, for each of the number of potentially relevant data assets, an augmented query score indicating a likelihood that the potentially relevant data asset is relevant to and useful for the search query based on the query relevancy scores and the utility scores, and generating augmented search results for the search query based on the augmented query scores.

Augmenting search results based on relevancy and utility

Systems and methods for identifying one or more master tables of a datalake are described. A system may obtain a plurality of computer readable tables of a datalake (with each computer readable table including one or more features). The system may also group the plurality of computer readable tables into a plurality of groups based on a number of features of each computer readable table of the plurality of computer readable tables. The system may further generate, for each of one or more groups of the plurality of groups, one or more neighborhoods based on a similarity of features between computer readable tables of the group. The system may also identify, for each neighborhood, one or more master tables from the one or more computer readable tables of the group. The system may further provide an indication of one or more master tables identified in the datalake.

Processing of computer readable tables in a datalake

A computational device receives, from a user, a selection of a block, a subsystem or multiple blocks from a block diagram modeling environment, and presents, to the user, a set of parameters associated with the block, subsystem, or multiple blocks. The computational device receives, from the user, selection of a first subset of parameters from the set of parameters, and creates a customized dialog box by promoting the selected first subset of parameters to the customized dialog box, wherein the customized dialog box permits editing of values associated only with the first subset of parameters.

Growing community of inventors

Bengaluru, India

Saikiran Sri Thunuguntla