Microsoft Technology Licensing, LLC

Srikumar Rangarajan

Aleksandras Surna

Sandeep Lingam

Artem A Oks

Qingqing Zhou

Sarvesh Singh

Wey Guy

Ashit Gosalia

Cipri Clinciu

Per-Ake Larson

Michael James Zwilling

Hanumantha Rao Kodavalla

Erik Ismert

Nimish S Khanolkar

Hari Krishna Sunder

Ajay Kalhan

Alan D Halverson

Jose A Blakeley

Steven John Lindell

Stefano Stefani

Milind Madhukar Joshi

Gang He

F Soner Terek

Srikanth Avadhanam

Pravin Mittal

Marcel Van Der Holst

Tsvi M Reiter

Rishabh Singh Ahluwalia

Nagavamsi Ponnekanti

Tianhui Shi

Xianfei Deng

Donald Cha

Alan Dale Halverson

Methods for data visibility in nested transactions in distributed systems are performed by systems and devices. Distributed executions of queries are performed in processing systems according to isolation level protocols with unique nested transaction identifiers for data management and versioning across one or more data sets, one or more compute pools, etc., within a logical server via a single transaction manager that oversees the isolation semantics and data versioning. A distributed query processor of the systems and devices performs nested transaction versioning for distributed tasks by generating nested transaction identifiers, encoded in data rows, which are used to enforce correct data visibility. Data visibility is restricted to previously committed data from distributed transactions and tasks, and is blocked for distributed transactions and tasks that run concurrently. Local commits for completed transactions and tasks are used to minimize transaction manager interactions, and instant rollbacks are enabled for aborted transactions and tasks.

Data visibility for nested transactions in distributed systems

Distributed database systems including a plurality of SQL compute nodes are described herein that enable such nodes to operate with versioned metadata despite the fact that SQL is only single-version aware. The distributed database system further includes a global logical metadata server to store and manage versions of metadata, to determine which of such versions should be visible at any given point in time, and enable creation of a virtual database that includes the proper versions of metadata. In an aspect, a central transaction manager manages global transaction identifiers and their associated start times, abort times and/or commit times that enables determination of transaction and metadata version visibility for any point in time. In an aspect, the visible metadata is included in a virtual database that logically overlays a physical database and provides the correct version of metadata in lieu of the current metadata version stored in the physical database.

Versioned metadata using virtual databases

Methods for snapshot isolation query transactions in distributed systems are performed by systems and devices. Distributed executions of queries are performed in a processing system according to an isolation level protocol for data management and data versioning across one or more data sets, one or more compute pools, etc., within a logical server via a single transaction manager that oversees the isolation semantics and data versioning. Read transactions of queries are performed lock-free via the isolation semantics, and instant rollbacks, point-in-time queries, single-phase commits in the distributed systems are also provided. Abort and clean up operations are performed based on a distributed abort protocol and a determined oldest active transaction for the system in which the single transaction manager does not track read-only transactions, and client nodes do not maintain commit tables for transactions.

Snapshot isolation query transactions in distributed systems

Methods for snapshot isolation query transactions in distributed systems are performed by systems and devices. Distributed executions of queries are performed in a processing system according to an isolation level protocol for data management and data versioning across one or more data sets, one or more compute pools, etc., within a logical server via a single transaction manager that oversees the isolation semantics and data versioning. Read transactions of queries are performed lock-free via the isolation semantics, and instant rollbacks, point-in-time queries, single-phase commits in the distributed systems are also provided. Abort and cleanup operations are performed based on a distributed abort protocol and a determined oldest active transaction for the system in which the single transaction manager does not track read-only transactions, and client nodes do not maintain commit tables for transactions.

Methods for rowgroup consolidation with delta accumulation and versioning in distributed systems are performed. The systems provide performant methods of row storage that enable versioned modifications of data while keeping and allowing access to older versions of the data for point-in-time transactions. The accumulation of valid rows, deletes, and modifications is maintained in blobs for rowgroups until a size threshold is reached, at which point the rows are moved into a columnar compressed form. Changes to data and associated metadata are stored locally and globally via appends, maintaining logical consistency. Metadata is stored in footers of files allowing faster access to the metadata and its associated data for transactions and instant rollback via metadata version flipping for aborted transactions, as well as lock-free reads of data.

Rowgroup consolidation with global delta accumulation and versioning in distributed systems

Architecture that includes an index creation algorithm that utilizes available resources and dynamically adjusts to successfully scale with increased resources and be able to do so for any data distribution. The resources can be processing resources, memory, and/or input/output, for example. A finer level of granularity, called a segment, is utilized to process tuples in a partition while creating an index. The segment also aligns with compression techniques for the index. By choosing an appropriate size for a segment and using load balancing the overall time for index creation can be reduced. Each segment can then be processed by a single thread thereby limiting segment skew. Skew is further limited by breaking down the work done by a thread into parallelizable stages.

Scalable index build techniques for column stores

Methods, systems, and computer-readable media of columnar storage of a database index are disclosed. A particular columnar index includes a column store that stores rows of the columnar index in a column-wise fashion and a delta store that stores rows of the columnar index in a row-wise fashion. The column store also includes an absence flag array. The absence flag array includes entries that indicate whether certain rows have been logically deleted from the column store.

Columnar storage of a database index

A data normalization system is described herein that represents multiple data types that are common within database systems in a normalized form that can be processed uniformly to achieve faster processing of data on superscalar CPU architectures. The data normalization system includes changes to internal data representations of a database system as well as functional processing changes that leverage normalized internal data representations for a high density of independently executable CPU instructions. Because most data in a database is small, a majority of data can be represented by the normalized format. Thus, the data normalization system allows for fast superscalar processing in a database system in a variety of common cases, while maintaining compatibility with existing data sets.

Normalizing data for fast superscalar processing

Growing community of inventors

Sammamish, WA, United States of America

Srikumar Rangarajan