Microsoft Technology Licensing, LLC

Maninderjit Singh Parmar

Edgars Sedols

Babatunde Micheal Okutubo

Embodiments described herein are directed to detecting data inconsistencies within a distributed database and identifying the cause thereof. For example, lineage events are emitted from different components of the distributed system that operate on various data files. A consistency checking engine analyzes these events and detects inconsistencies with respect to the data files. The embodiments described herein checks the integrity of the database and assists in understanding the root cause in case of a corruption. Moreover, it provides the timeline for the corruption and whether it is repairable or not. These properties enable determining the right time to restore the customer's database or the right set of actions to repair the corruption. In case of repairable corruption, the correct compensating repair actions may be applied.

Consistency checking for distributed analytical database systems

Methods and systems are provided for improved access to rows of data in a distributed data system. Each data row is associated with a partition. Data rows are distributed in one or more files and an impure file includes data rows associated multiple partitions. A clustering set is generated from a plurality of impure files by selecting a candidate impure file based on file access activity metrics and one or more neighbor impure files. Data rows of the impure files included in the clustering set are sorted according to their respective associated partitions. A set of disjoint partition range files are generated based on the sorted data rows of the impure files included in the clustering set. Each file of the set of disjoint partition range files is transferred to a respective target partition.

Growing community of inventors

Redmond, WA, United States of America

Maninderjit Singh Parmar