Microsoft Technology Licensing, LLC

Ju Wang

Bradley Gene Calder

Arild Einar Skjolsvold

Shane Kumar Mainali

Hemal Khatri

Andrew James Edwards

Shashwat Srivastav

Niranjan Nilakantan

Krishnan Varadarajan

Aaron William Ogus

Jiesheng Wu

Jegan Devaraju

Sriram Sankaran

Padmanabha Chakravarthy Uddaraju

Yang Yang Zhang

Abdul Rafay Abbasi

Manish Chablani

Pradeep Kumar Gunda

Kavitha Manivannan

Marvin McNett, Ii

Shyam Antony

Venkates Paramasivam Balakrishnan

Quan Zhang

Hao Feng

Vaman Bedekar

Samuel James McKelvie

Yikang Xu

Deepali Bhardwaj

Xinran Wu

Cheng Huang

Maneesh Sah

Ivan Brugiolo

Samer Arafeh

Huseyin Simitci

Ioan Oltean

Yue Zuo

Anu Engineer

Ovais Khan

Harshawardhan Gadgil

Manan Shah

Sivakumar Kalva

Sridhar Srinivasan

Xinhua Ji

Maxim Mazeev

Zichen Sun

Shaoyu Zhang

Alexander Felsöbüki Nagy

He Su

Leonidas Rigas

Ralph Burton Harris, Iii

Jianwu Gao

Kaifeng Zhu

Hosung Song

Hakon Gabriel Verespej

Bin Tong

Sriram Krishnan

Muhammad Junaid Shahid

Zhe Yang

Pradeep Seela

Vikram Dhaneshwar

Alexander Felsobuki Nagy

Ziehen Sun

Vanam Bedekar

Arlid E Skjolsvold

Yang Zhang

Marvin Mcnett Ii

Ralph Burton Harris Iii

Samuel James Mckelvie

Jegan Devanraju

A distributed storage system includes table controllers collectively storing partitions of a logical table having multiple rows. For each of the partitions, a corresponding table controller stores a subset of the rows of the logical table across individual pages. A management controller updates a schema of the logical table from a first to a second schema definition by, for each of the partitions, sending an update command indicating the second schema definition to the corresponding table controller. The first table controller corresponds to a first partition that includes a first subset of the rows. The first table controller, prior to receiving the update command, includes an identifier of the first schema definition within each page of data for the first subset. Subsequent to receiving the update command, the first table controller, when writing each new page of data for the first subset, includes an identifier of the second schema definition.

Online schema change of range-partitioned index in a distributed storage system

Methods, systems, apparatuses, and computer program products are provided that enable immediate access to data associated with a source blob. The foregoing is achieved by creating a copy-on-read blob based on the source blob and a link blob based on the copy-on-read blob for each virtual machine to be deployed. When creating the copy-on-read blob, the source blob is not copied. Instead, data from the source blob is copied to the copy-on-read blob upon request by a virtual machine. Subsequent requests for that data are retrieved from the copy-on-read blob. Each link blob stores write data provided by its corresponding virtual machine. When a virtual machine requests data, its corresponding link blob is checked for the data. If the data is not stored therein, the copy-on-read blob is checked for the data. If the copy-on-read blob does not store the data, the data is retrieved from the source blob.

Providing instant and distributed access to a source blob via copy-on-read blobs and link blobs

Systems and methods are provided for assigning and associating resources in a cloud computing environment. Virtual machines in the cloud computing environment can be assigned or associated with pools corresponding to users as dedicated, standby, or preemptible machines. The various states provide users with the ability to reserve a desired level of resources while also allowing the operator of the cloud computing environment to increase resource utilization.

Assignment of resources in virtual machine pools

Systems and methods are provided that enable a general framework for partitioning application-defined jobs in a scalable environment. The general framework decouples partitioning of a job from the other aspects of the job. As a result, the effort required to define the application-defined job is reduced or minimized, as the user is not required to provide a partitioning algorithm. The general framework also facilitates management of masters and servers performing computations within the distributed environment.

Managing partitions in a scalable environment

Anchor tree cross-merge within a distributed storage system. A computer system identifies a data structure that includes (i) a root anchor tree and (ii) an ordered set of anchor trees that are ordered based on their creation, such that a last anchor tree of the ordered set of anchor trees is most-recently created. The data structure stores index data for a set of objects stored in a non-volatile storage. The computer system creates a new anchor tree in the ordered set of anchor trees. The computer system identifies, from a set of delta tables, index data representing one or more objects that are stored on the non-volatile storage. The computer system merges the index data representing the one or more objects into the new anchor tree.

Staging anchor trees for improved concurrency and performance in page range index management

Presented herein are methods of replicating versioned and hierarchical data structures, as well as data structures representing complex transactions. Due to interdependencies between data entities and a lack of guaranteed message ordering, simple replication methods employed for simple data types cannot be used. Operations on data structures exhibit dependencies between the messages making up the operations. This strategy can be extended to various types of complex transactions by considering certain messages to depend on other messages or on the existence of other entries at the data store. Regardless of origin, these dependencies can be enforced by suspending the processing of messages with unsatisfied dependencies until all of its dependencies have been met. Alternately, transactions can be committed immediately, creating entities that include versioned identifiers for each of their dependencies. These entities can then be garbage collected of the parent objects are not subsequently created.

Versioned and hierarchical data structures and distributed transactions

A distributed storage system includes non-volatile storage storing portions of a first object. The first object encompasses data having a first range of addresses and each portion includes data for a respective range of addresses that is a proper subset of the first range. A first data structure stores, for each portion, data indicating the respective range of addresses and a pointer to where the portion is stored. The first data structure includes a root tree and a set of trees ordered by creation data such that a last tree is most-recently created. The non-volatile storage stores received write data and a write buffer stores index data pointing to storage locations of the received write data. An index management system stores the index data from the write buffer into the last tree and, if the ordered set is empty, creates a tree in the ordered set before the storing.

In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata. The bootstrap layer is used to store the extent table as a system table and to persist the state of the stream manager as system streams.

Distributed metadata management in a distributed storage system

Embodiments provide a method to collect aggregate information or usage data quickly and efficiently with minimal lag. Additionally, the system can use this aggregate information internally for improved load balancing, better data placement, optimization, and enhanced debugging. The system can quickly look at aggregate information across a huge amount of data and drill down cheaply because the aggregate information is generated using existing processes. Aggregated statistics storage and collection may be built on top of an LSM tree used to store a persistent index for a cloud storage system. The statistics may also represent the result of an operation (e.g., max, min, sum, average) on selected parameter(s) or attribute(s) of stored data. Aggregate statistics values may be efficiently maintained during index merge and garbage collection processes or any other index management. As delta LSM trees are merged into a base LSM tree, the aggregates are updated in delta fashion.

File table index aggregate statistics

In various embodiments, methods and systems for implementing garbage collection in distributed storage systems are provided. The distributed storage system operates based on independent management of metadata of extent and stream data storage resources. A hybrid garbage collection system based on reference counting garbage collection operations and mark-and-sweep garbage collection operations is implemented. An extent lifetime table that tracks reference weights and mark sequences for extents is initialized and updated based on indications from extent managers and stream managers, respectively. Upon determining that an extent is to be handed-off from weighted reference counting garbage collection operations to mark-and-sweep garbage collection operations, a reference weight field for the extent is voided and a mark sequence field of the extent is updated. The mark sequence field is updated with a latest global sequence number. The mark-and-sweep garbage collection operations are utilized to reclaim the extent when the extent is no longer referenced.

Hybrid garbage collection in a distributed storage system

Growing community of inventors

Redmond, WA, United States of America

Ju Wang