Microsoft Technology Licensing, LLC

Jegan Devaraju

Maneesh Sah

Shane Kumar Mainali

Krishnan Varadarajan

Ju Wang

Manish Chablani

Quan Zhang

Hao Feng

Arild Einar Skjolsvold

Rushi Srinivas Surla

Zichen Sun

Yongfu Lou

Venkates Paramasivam Balakrishnan

Xinhua Ji

Sandesh Doddameti

Pradeep Seela

Sridhar Srinivasan

Andrew James Edwards

Shaoyu Zhang

Himanshu Gupta

Yi-Fan Tsai

Jianwu Gao

Pei Zhang

Kristopher T Lange

Kaifeng Zhu

Bin Tong

Bradley Gene Calder

Aaron William Ogus

Yikang Xu

Georgi M Chalakov

Ioan Oltean

He Su

Muhammad Junaid Shahid

Michael E Roberson

James Lee Baker

Narayanamurthy Sudhakar

Julia Johnstone

Ramy Wassef

Vamshi Konagari

Hao Wang

Donglin Wei

Qiuhan Wang

Ziehen Sun

Advait Kumar Mishra

A distributed storage system includes table controllers collectively storing partitions of a logical table having multiple rows. For each of the partitions, a corresponding table controller stores a subset of the rows of the logical table across individual pages. A management controller updates a schema of the logical table from a first to a second schema definition by, for each of the partitions, sending an update command indicating the second schema definition to the corresponding table controller. The first table controller corresponds to a first partition that includes a first subset of the rows. The first table controller, prior to receiving the update command, includes an identifier of the first schema definition within each page of data for the first subset. Subsequent to receiving the update command, the first table controller, when writing each new page of data for the first subset, includes an identifier of the second schema definition.

Online schema change of range-partitioned index in a distributed storage system

Described herein is a system and method for an append-only block-oriented object storage system. An unstructured storage object is stored in cloud storage. A block is data is received, from a client, to append to the unstructured storage object. A universally unique identifier of the client, and client contextual information regarding the block of data, are received from the client. The client contextual information comprises an identifier of a most recently appended block from the client. When the identifier of the most recently appended block from the client matches a stored identifier associated with the universally unique identifier of the client, the block of data is appended to the unstructured storage object, and the stored identifier associated with the universally unique identifier of the client is updated in accordance with the received client contextual information.

Object storage change-events

Anchor tree cross-merge within a distributed storage system. A computer system identifies a data structure that includes (i) a root anchor tree and (ii) an ordered set of anchor trees that are ordered based on their creation, such that a last anchor tree of the ordered set of anchor trees is most-recently created. The data structure stores index data for a set of objects stored in a non-volatile storage. The computer system creates a new anchor tree in the ordered set of anchor trees. The computer system identifies, from a set of delta tables, index data representing one or more objects that are stored on the non-volatile storage. The computer system merges the index data representing the one or more objects into the new anchor tree.

Staging anchor trees for improved concurrency and performance in page range index management

Example aspects include techniques for ordering operations in a file system having a hierarchical namespace including determining, by a first service that manages file system operations for a first entity in the hierarchical namespace, to perform a file system operation for the first entity, generating, by the first service, a first counter record for the file system operation, obtaining, based on determining to perform the file system operation and from a second service that manages file system operations on a second entity that is a parent of the entity in the hierarchical namespace, a second counter record for a last file system operation performed on the second entity, wherein the second counter record is generated by the second service, and storing, in a change feed, an indication of the file system operation along with the first counter record and the second counter record.

Methods and systems for ordering operations on a file system having a hierarchical namespace

Techniques are provided for managing resources among clusters of computing devices in a computing system. Resource reassignment message are generated for indicating that servers are reassigned and in response to resource compute loads exceed or fall below certain thresholds. Techniques also include establishing communications with the reassigned servers to assign compute loads without physically relocating the servers from one cluster to another cluster.

Cluster resource management in distributed computing systems

A distributed storage system includes non-volatile storage storing portions of a first object. The first object encompasses data having a first range of addresses and each portion includes data for a respective range of addresses that is a proper subset of the first range. A first data structure stores, for each portion, data indicating the respective range of addresses and a pointer to where the portion is stored. The first data structure includes a root tree and a set of trees ordered by creation data such that a last tree is most-recently created. The non-volatile storage stores received write data and a write buffer stores index data pointing to storage locations of the received write data. An index management system stores the index data from the write buffer into the last tree and, if the ordered set is empty, creates a tree in the ordered set before the storing.

Described herein is a system and method for a user accessible log of object storage change-events system. A change is detected within a cloud storage based on an unstructured storage object stored in the cloud storage. Contextual attributes of the change are identified from an updated state of the log file that stores information about the unstructured storage object. An output is provided based, at least in part, upon the detected change and at least some of the contextual attributes, wherein the output is an immutable, read-only log of change events.

Techniques for managing resources among clusters of computing devices in a computing system are described herein. In one embodiment, a method includes receiving, via a computer network, a resource reassignment message indicating that a server is reassigned from a first cluster to a second cluster and in response to the received resource reassignment message, establishing communications with the server reassigned from the first cluster to the second cluster via the computer network. The method further includes subsequent to establishing communications with the server via the computer network, assigning a compute load to the server reassigned from the first cluster to the second cluster without physically relocating the server from the first cluster to the second cluster.

Data is replicated between primary and secondary storage systems using a data pulling process or a log shipping process. If data is to be replicated through the log shipping process, a transaction and its associated data get stored in a transaction log in the secondary storage system. Subsequently, when the transaction gets executed, actual data gets persisted from the transaction log to a data log in the secondary storage system. If the data is to be replicated through the data pulling process, the transaction is stored in the transaction log. However, the associated data for the transaction is retrieved from the first primary storage system and stored directly in the data log in the secondary system.

Data replication in a distributed storage system

A snapshot of data from a table associated with a particular user may be generated. Tree data structures that are distributed across multiple computer systems may be accessed. Each of the tree structures may include data associated with one or more users. At least one tree data structure of the tree data structures that includes data associated with the particular user of the one or more users may be identified. The at least one tree data structure may then be filtered. Filtering may comprise identifying only data that is associated with the particular user. A snapshot of the data associated with the particular user may be generated. Generating the snapshot of the data associated with the particular user comprises generating a data structure that is configured to map to each data page of the at least one tree data structure that includes data associated with the particular user.

Efficient snapshot generation of data tables

Embodiments provide a method to collect aggregate information or usage data quickly and efficiently with minimal lag. Additionally, the system can use this aggregate information internally for improved load balancing, better data placement, optimization, and enhanced debugging. The system can quickly look at aggregate information across a huge amount of data and drill down cheaply because the aggregate information is generated using existing processes. Aggregated statistics storage and collection may be built on top of an LSM tree used to store a persistent index for a cloud storage system. The statistics may also represent the result of an operation (e.g., max, min, sum, average) on selected parameter(s) or attribute(s) of stored data. Aggregate statistics values may be efficiently maintained during index merge and garbage collection processes or any other index management. As delta LSM trees are merged into a base LSM tree, the aggregates are updated in delta fashion.

File table index aggregate statistics

In various embodiments, methods and systems for implementing hash-based partitioning in distributed computing systems are provided. At a high level, a distributed computing system having an underlying range-based partitioning architecture for storage may be configured as a hash-based partitioning system, for example, a hybrid range-hash table storage. An operations engine of the hash-based partitioning system receives a tenant request to provision input/output operations per second (IOPS). The tenant request comprises a requested number of IOPS. Based on the tenant request, a provisioning operation to provision IOPS in a hybrid range-hash table storage with hash-based partitioning is determined. The provisioning operation is selected from one of the following: a table creation provisioning operation, an IOPS increase provisioning operation, and an IOPS decrease provisioning operation. The selected provisioning operation is executed for a corresponding table. A user request for data is processed using the table associated with the requested number of IOPS.

Hash-based partitioning system

A distributed storage system includes multiple partitions. A block received for storage is striped across the partitions if it meets predetermined criteria. Multiple tables are maintained for indexing the blocks and the subblocks of blocks that are partitioned. The tables for the subblocks and the tables for the corresponding blocks of the subblocks are updated to include metadata for the subblocks for retrieving the subblocks in response to a request for a corresponding stored block.

Data partitioning in a distributed storage system

Live migration of partitions can be achieved using server-to-server communications to coordinate and control partition migration and rebuild a persistent state of the partition on a destination server while the partition is served on the source server. The live migration approach breaks the heavy dependency on the partition master by eliminating most of the server-to-master communications, delegating coordination and control of the migration to the source and destination servers. After the migration is initiated by the partition master, the source server sends in-memory cache entries indicating the persistent state of the partition to the destination server. Once transferred, the two servers can cooperate to complete the handover, which loads the partition on the destination server using the cache entries received from the source server. Once the source server offloads the partition, it can redirect traffic for the partition to the destination server by triggering an update to the partition map.

Live migration of partitions

Growing community of inventors

Redmond, WA, United States of America

Jegan Devaraju