Microsoft Technology Licensing, LLC

Sandeep Lingam

Sameer Arun Verkhedkar

Steven John Lindell

Srikumar Rangarajan

Girish Mittur Venkataramanappa

Zhengguo Sun

Varun Kunjbihari Tibrewal

Colin Neville

Sarvesh Singh

Hari Krishna Sunder

Cristian Diaconu

Tomas Talius

Alan D Halverson

Alexander Evgenievich Nosov

Naveen Prakash

Alejandro Hernandez Saenz

Peter Byrne

Alexander Budovski

Rogerio Ramos

Krystyna Ewa Reisteter

Huanhui Hu

Daniel L Lemmond

Rishabh Singh Ahluwalia

Tianhui Shi

Xianfei Deng

Donald Cha

Alan Dale Halverson

Methods for data visibility in nested transactions in distributed systems are performed by systems and devices. Distributed executions of queries are performed in processing systems according to isolation level protocols with unique nested transaction identifiers for data management and versioning across one or more data sets, one or more compute pools, etc., within a logical server via a single transaction manager that oversees the isolation semantics and data versioning. A distributed query processor of the systems and devices performs nested transaction versioning for distributed tasks by generating nested transaction identifiers, encoded in data rows, which are used to enforce correct data visibility. Data visibility is restricted to previously committed data from distributed transactions and tasks, and is blocked for distributed transactions and tasks that run concurrently. Local commits for completed transactions and tasks are used to minimize transaction manager interactions, and instant rollbacks are enabled for aborted transactions and tasks.

Data visibility for nested transactions in distributed systems

Distributed database systems including a plurality of SQL compute nodes are described herein that enable such nodes to operate with versioned metadata despite the fact that SQL is only single-version aware. The distributed database system further includes a global logical metadata server to store and manage versions of metadata, to determine which of such versions should be visible at any given point in time, and enable creation of a virtual database that includes the proper versions of metadata. In an aspect, a central transaction manager manages global transaction identifiers and their associated start times, abort times and/or commit times that enables determination of transaction and metadata version visibility for any point in time. In an aspect, the visible metadata is included in a virtual database that logically overlays a physical database and provides the correct version of metadata in lieu of the current metadata version stored in the physical database.

Versioned metadata using virtual databases

Methods for snapshot isolation query transactions in distributed systems are performed by systems and devices. Distributed executions of queries are performed in a processing system according to an isolation level protocol for data management and data versioning across one or more data sets, one or more compute pools, etc., within a logical server via a single transaction manager that oversees the isolation semantics and data versioning. Read transactions of queries are performed lock-free via the isolation semantics, and instant rollbacks, point-in-time queries, single-phase commits in the distributed systems are also provided. Abort and clean up operations are performed based on a distributed abort protocol and a determined oldest active transaction for the system in which the single transaction manager does not track read-only transactions, and client nodes do not maintain commit tables for transactions.

Snapshot isolation query transactions in distributed systems

Methods for snapshot isolation query transactions in distributed systems are performed by systems and devices. Distributed executions of queries are performed in a processing system according to an isolation level protocol for data management and data versioning across one or more data sets, one or more compute pools, etc., within a logical server via a single transaction manager that oversees the isolation semantics and data versioning. Read transactions of queries are performed lock-free via the isolation semantics, and instant rollbacks, point-in-time queries, single-phase commits in the distributed systems are also provided. Abort and cleanup operations are performed based on a distributed abort protocol and a determined oldest active transaction for the system in which the single transaction manager does not track read-only transactions, and client nodes do not maintain commit tables for transactions.

Methods for rowgroup consolidation with delta accumulation and versioning in distributed systems are performed. The systems provide performant methods of row storage that enable versioned modifications of data while keeping and allowing access to older versions of the data for point-in-time transactions. The accumulation of valid rows, deletes, and modifications is maintained in blobs for rowgroups until a size threshold is reached, at which point the rows are moved into a columnar compressed form. Changes to data and associated metadata are stored locally and globally via appends, maintaining logical consistency. Metadata is stored in footers of files allowing faster access to the metadata and its associated data for transactions and instant rollback via metadata version flipping for aborted transactions, as well as lock-free reads of data.

Rowgroup consolidation with global delta accumulation and versioning in distributed systems

Embodiments described herein detect data corruption in a distributed data set system. For example, a system comprises node(s) for processing queries with respect to a distributed data set comprising a plurality of storage segments. A write transaction resulting from a query with respect to a particular storage segment is logged in a log record that describes a modification to the storage segment. A log service provides the log record to a data server managing a portion of the distributed data set in which the storage segment is included, which performs the write transaction with respect to the storage segment. For redundancy purposes, the data server has replica(s) that manage respective replicas of the portion of the distributed data set managed thereby. For backup purposes, snapshots of the replica(s) are periodically generated. To determine a data corruption, a snapshot of one replica is cross-validated with a snapshot of another replica.

Snapshot-based data corruption detection

A Distributed Availability Group (DAG) spans two AGs, each spanning one or more replica nodes and functioning as primary or secondary AG. A primary AG is replicated to the secondary AG synchronously or asynchronously. A failover in the DAG results in the AGs swapping their roles. Multiple DAGs can be linked together as a chain, which provides many useful features including disaster recovery across geographical regions, massive read scale (numerous readable secondary nodes), online migration of databases (across different operating systems and computing environments). The systems using DAGs can replicate databases across multiple independent high availability (HA) failover clusters using complex replication topologies and allow for manual failover and failback. The systems allow chaining of multiple AGs to provision a treelike structure of replicas and numerous secondary replicas without impacting performance. The systems automatically seed new database replicas to facilitate building a complex topology of DAGs.

Distributed availability groups of databases for data centers including seeding, synchronous replications, and failover

Application service configuration of a timeframe for performing dataloss failover (failover that does not attempt full data replication to the secondary data store) from a primary data store to the secondary data store. A data-tier service, such as perhaps a database as a service (or DBaaS), could receive that configuration from the application service and automatically perform the dataloss failover as configured by the application service. This relieves the application service from having to manage the failover workflow while still allowing the application service to appropriately balance the timing of dataloss failover, which will depend on a very application-specific optimal balance between the negative effects of operational latency versus dataloss.

Application service-level configuration of dataloss failover

Distributed availability groups of databases for data centers for providing massive read scale

Distributed availability groups of databases for data centers including different commit policies

A Distributed Availability Group (DAG) spans two AGs, each spanning one or more replica nodes and functioning as primary or secondary AG. A primary AG is replicated to the secondary AG synchronously or asynchronously. A failover in the DAG results in the AGs swapping their roles. Multiple DAGs can be linked together as a chain, which provides many useful features including disaster recovery across geographical regions, massive read scale (numerous readable secondary nodes), online migration of databases (across different operating systems and computing environments). The systems using DAGs can replicate databases across multiple independent high availability (HA) failover clusters using complex replication topologies and allow for manual failover and failback. The systems allow chaining of multiple AGs to provision a treelike structure of replicas and numerous secondary replicas without impacting performance. The systems automatically seed new database replicas that to facilitate building a complex topology of DAGs.

Distributed availability groups of databases for data centers

Distributed availability groups of databases for data centers including failover to regions in different time zones

Growing community of inventors

Redmond, WA, United States of America

Sandeep Lingam