Microsoft Technology Licensing, LLC

Krishnan Varadarajan

Shane Kumar Mainali

Venkates Paramasivam Balakrishnan

Ju Wang

Jegan Devaraju

Maneesh Sah

Michael L Chu

Quan Zhang

Manish Chablani

Andrew James Edwards

Zichen Sun

Hao Feng

Ivan Brugiolo

Ovais Khan

Manan Shah

Sivakumar Kalva

Sridhar Srinivasan

Shaoyu Zhang

He Su

Pradeep Seela

Jianwu Gao

Kaifeng Zhu

Bin Tong

Bradley Gene Calder

Aaron William Ogus

Henry Gabryjelski

Donald James McCrady

Peter W Wieland

Yikang Xu

Genevieve Fernandes

Malcolm James Smith

Marko Radmilac

Georgi M Chalakov

Raju Ramanathan

Sundar P Subramani

Muhammad Junaid Shahid

Yongfu Lou

Michael E Roberson

Jieqing Wang

Pei Zhang

Yi-Fan Tsai

Bitan Deb

Kristopher T Lange

Malavika Arunmozhi

Ziehen Sun

Vijay Raghunathan

Bharat Joshi

Advait Kumar Mishra

Jegan Devanraju

A method, computer program product, and computing system for processing a first data access request from a first computing device for accessing a cloud storage portion of a plurality of cloud storage portions associated with a cloud-based storage resource. A current ownership sequence identifier associated with the cloud storage portion is obtained. The current ownership sequence identifier from the cloud storage portion is compared to a previously obtained ownership sequence identifier for the cloud storage portion on the first computing device. The first data access request is effectuated on the cloud storage portion when the current ownership sequence identifier is identical to the previously obtained ownership sequence identifier.

System and method for managing cloud-based data ownership using access requests

Various methods and systems for implementing request scheduling and processing in a multi-tenant distributed computing environment are provided. Requests to utilize system resources in the distributed computing environment are stored in account queues corresponding to tenant accounts. If storing a request in an account queue would exceed a throttling threshold such as a limit on the number of requests stored per account, the request is dropped to a throttling queue. A scheduler prioritizes processing requests stored in the processing queue before processing requests stored in the account queues. The account queues can be drained using dominant resource scheduling. In some embodiments, a request is not picked up from an account queue if processing the request would exceed a predefined hard limit on system resource utilization for the corresponding tenant account. In some embodiments, the hard limit is defined as a percentage of threads the system has to process requests.

Throttling queue for a request scheduling and processing system

A distributed storage system includes table controllers collectively storing partitions of a logical table having multiple rows. For each of the partitions, a corresponding table controller stores a subset of the rows of the logical table across individual pages. A management controller updates a schema of the logical table from a first to a second schema definition by, for each of the partitions, sending an update command indicating the second schema definition to the corresponding table controller. The first table controller corresponds to a first partition that includes a first subset of the rows. The first table controller, prior to receiving the update command, includes an identifier of the first schema definition within each page of data for the first subset. Subsequent to receiving the update command, the first table controller, when writing each new page of data for the first subset, includes an identifier of the second schema definition.

Online schema change of range-partitioned index in a distributed storage system

Methods, systems, apparatuses, and computer program products are provided that enable immediate access to data associated with a source blob. The foregoing is achieved by creating a copy-on-read blob based on the source blob and a link blob based on the copy-on-read blob for each virtual machine to be deployed. When creating the copy-on-read blob, the source blob is not copied. Instead, data from the source blob is copied to the copy-on-read blob upon request by a virtual machine. Subsequent requests for that data are retrieved from the copy-on-read blob. Each link blob stores write data provided by its corresponding virtual machine. When a virtual machine requests data, its corresponding link blob is checked for the data. If the data is not stored therein, the copy-on-read blob is checked for the data. If the copy-on-read blob does not store the data, the data is retrieved from the source blob.

Providing instant and distributed access to a source blob via copy-on-read blobs and link blobs

Anchor tree cross-merge within a distributed storage system. A computer system identifies a data structure that includes (i) a root anchor tree and (ii) an ordered set of anchor trees that are ordered based on their creation, such that a last anchor tree of the ordered set of anchor trees is most-recently created. The data structure stores index data for a set of objects stored in a non-volatile storage. The computer system creates a new anchor tree in the ordered set of anchor trees. The computer system identifies, from a set of delta tables, index data representing one or more objects that are stored on the non-volatile storage. The computer system merges the index data representing the one or more objects into the new anchor tree.

Staging anchor trees for improved concurrency and performance in page range index management

A distributed storage system includes non-volatile storage storing portions of a first object. The first object encompasses data having a first range of addresses and each portion includes data for a respective range of addresses that is a proper subset of the first range. A first data structure stores, for each portion, data indicating the respective range of addresses and a pointer to where the portion is stored. The first data structure includes a root tree and a set of trees ordered by creation data such that a last tree is most-recently created. The non-volatile storage stores received write data and a write buffer stores index data pointing to storage locations of the received write data. An index management system stores the index data from the write buffer into the last tree and, if the ordered set is empty, creates a tree in the ordered set before the storing.

Described herein is a system method for point in time blob restore. In response to receipt of a request to restore a particular blob range to a particular point in time, read and/or write operations on the particular blob range requested to be restored are blocked. Blob(s) in the particular blob range which have changed since the particular point in time are determined based, at least in part, upon information stored in a change log and/or a deleted blob store. A determination is made as to whether the requested restore operation can be performed, for example, based upon a restoration policy. When it is determined that the requested restore operation can be performed, blob(s) determined to have changed since the particular point in time based, at least in part, upon information stored in the change log and/or the deleted blob store are restored.

Point in time blob restore

Data is replicated between primary and secondary storage systems using a data pulling process or a log shipping process. If data is to be replicated through the log shipping process, a transaction and its associated data get stored in a transaction log in the secondary storage system. Subsequently, when the transaction gets executed, actual data gets persisted from the transaction log to a data log in the secondary storage system. If the data is to be replicated through the data pulling process, the transaction is stored in the transaction log. However, the associated data for the transaction is retrieved from the first primary storage system and stored directly in the data log in the secondary system.

Data replication in a distributed storage system

A snapshot of data from a table associated with a particular user may be generated. Tree data structures that are distributed across multiple computer systems may be accessed. Each of the tree structures may include data associated with one or more users. At least one tree data structure of the tree data structures that includes data associated with the particular user of the one or more users may be identified. The at least one tree data structure may then be filtered. Filtering may comprise identifying only data that is associated with the particular user. A snapshot of the data associated with the particular user may be generated. Generating the snapshot of the data associated with the particular user comprises generating a data structure that is configured to map to each data page of the at least one tree data structure that includes data associated with the particular user.

Efficient snapshot generation of data tables

Embodiments provide a method to collect aggregate information or usage data quickly and efficiently with minimal lag. Additionally, the system can use this aggregate information internally for improved load balancing, better data placement, optimization, and enhanced debugging. The system can quickly look at aggregate information across a huge amount of data and drill down cheaply because the aggregate information is generated using existing processes. Aggregated statistics storage and collection may be built on top of an LSM tree used to store a persistent index for a cloud storage system. The statistics may also represent the result of an operation (e.g., max, min, sum, average) on selected parameter(s) or attribute(s) of stored data. Aggregate statistics values may be efficiently maintained during index merge and garbage collection processes or any other index management. As delta LSM trees are merged into a base LSM tree, the aggregates are updated in delta fashion.

File table index aggregate statistics

In various embodiments, methods and systems for implementing hash-based partitioning in distributed computing systems are provided. At a high level, a distributed computing system having an underlying range-based partitioning architecture for storage may be configured as a hash-based partitioning system, for example, a hybrid range-hash table storage. An operations engine of the hash-based partitioning system receives a tenant request to provision input/output operations per second (IOPS). The tenant request comprises a requested number of IOPS. Based on the tenant request, a provisioning operation to provision IOPS in a hybrid range-hash table storage with hash-based partitioning is determined. The provisioning operation is selected from one of the following: a table creation provisioning operation, an IOPS increase provisioning operation, and an IOPS decrease provisioning operation. The selected provisioning operation is executed for a corresponding table. A user request for data is processed using the table associated with the requested number of IOPS.

Hash-based partitioning system

Growing community of inventors

Redmond, WA, United States of America

Krishnan Varadarajan