Microsoft Technology Licensing, LLC

Rogerio Ramos

Cristian Diaconu

Arkadi Brjazovski

Raghavendra Thallam Kodandaramaih

Christian Damianidis

Naveen Prakash

Alejandro Hernandez Saenz

Sandeep Lingam

Daniel Vasquez Lopez

Peter Byrne

Karthick Krishnamoorthy

Alexander Budovski

Krystyna Ewa Reisteter

Huanhui Hu

Jacob Rubin Lorch

Youssef M Barakat

Shirish Gajera

Fayssal Martani

Methods, systems, apparatuses, and computer program products are provided for coordinating a distributed database transaction. A transaction driver, such as a client machine, may initiate a distributed transaction. The transaction driver may transmit to a transaction coordinator a driver report that includes identifying information related to the distributed transaction, including an identification of participants involved in the transaction. The coordinator may determine whether participant reports, which include a status of the portion of the distributed database transaction of a particular participant, are received from each of the participants. Participant reports may also identify participants that are descendants of the reporting participant. The transaction coordinator may store, in a table, information to track the progress of the distributed transaction. Using the table, an outcome of the distributed transaction may be determined and transmitted to one or more of the involved participants.

Lock free distributed transaction coordinator for in-memory database participants

Embodiments described herein detect data corruption in a distributed data set system. For example, a system comprises node(s) for processing queries with respect to a distributed data set comprising a plurality of storage segments. A write transaction resulting from a query with respect to a particular storage segment is logged in a log record that describes a modification to the storage segment. A log service provides the log record to a data server managing a portion of the distributed data set in which the storage segment is included, which performs the write transaction with respect to the storage segment. For redundancy purposes, the data server has replica(s) that manage respective replicas of the portion of the distributed data set managed thereby. For backup purposes, snapshots of the replica(s) are periodically generated. To determine a data corruption, a snapshot of one replica is cross-validated with a snapshot of another replica.

Snapshot-based data corruption detection

Storing an incoming data stream using successive files that are consecutively populated. The appropriate file to populate a given data stream portion into is determined by mapping the data stream offset to a file, and potentially also an address within that file. The successive files may be the same size, so that the file can be identified based on the data stream address (or offset) without the use of an index. Furthermore, the files may be easily named by having that size be some multiple of a binary power of bytes. That way, the files themselves can be automatically and named and identified by using the more significant bit or bits of the data stream offset to uniquely identify the file and establish ordering of the files. Replication may occur from a primary to a secondary store by transmitting the offset, and the actual data to be stored.

Statelessly populating data stream into successive files

The insertion into a data stream of collection records associated with multiple collections of data items in the data stream, for purposes of aiding in subsequent processing of the data stream by collections. Each collection record comprises a collection definition that does not overlap with the collection definition in any of the sequence of collection records including the collection definition of any neighboring collection record in the sequence of collection records. Each collection record also includes a data stream address range extending from the first data item to the last data item in the collection.

Collection record for overlapping data stream collections

A lockless open collection data structure used to create a sequence of collection records. The sequence of collection records may be inserted into a data stream for purposes of assisting in subsequent processing of the data stream in a parallel manner, by collection. The open collection data structure is re-usable for multiple collections of data items within the data stream. The use of the open collection data structure may occur while a data stream that includes multiple different types of data items is being evaluated. The open collection data structure is used to track an expandable collection definition of a currently open collection. Upon detecting that the collection definition is no longer expandable, the collection record is created based on state within the open collection data structure. Once the collection record for a collection has been created, the open collection data structure becomes available for the next collection record.

Lockless open collection data structure

The logging of tasks associated with multiple transactions in the context in which the computing system is capable of having multiple of the transactions active at a given time, and thus in which the order of completion of the transactions is not guaranteed to be the same as the order of initiation of the transactions. A sequence of transaction segment records is added to the log. Each transaction segment record comprises a transaction identifier range that does not overlap with the transaction identifier range in any of the sequence of transaction segment records including the transaction identifier range of any neighboring transaction segment record in the sequence of transaction segment records. Each transaction segment record also includes a log entry identifier range extending from the first task of the first of the transactions to be initiated to the last task of the last transaction to complete.

Collection record location as log tail beginning

The use of a data stream that has therein data items and a sequence of collection records. each comprising a collection definition that is not overlapping with the collection definition in any of the sequence of collection records. The collection definition defines which data items of the data stream are included within the collection. Each collection record also includes a data stream address range at least extending from the data stream address of the first data item of the collection to the data stream address of the last data item in the collection. In this context, the data stream may be efficiently processed by quickly reviewing the data stream to find each collection record. Once a collection record is found, the collection record is dispatched for processing to a worker thread for processing of the corresponding collection.

Growing community of inventors

Redmond, WA, United States of America

Rogerio Ramos