Microsoft Technology Licensing, LLC

Quan Zhang

Shane Kumar Mainali

Zichen Sun

Krishnan Varadarajan

Shaoyu Zhang

Ju Wang

Maneesh Sah

Jegan Devaraju

Manish Chablani

Hao Feng

Jun Tse Chen

Thomas Leo Marquardt

Georgi M Chalakov

Jingchao Zhang

Junhua Gu

Amit Pratap Singh

Da Zhou

Saher B Ahwal

Dana Yulian Kaban

Saurabh Pant

Esfandiar Manii

Wei Zhu Chen

Venkates Paramasivam Balakrishnan

Sridhar Srinivasan

He Su

Kaifeng Zhu

Bin Tong

Jianwu Gao

Arild Einar Skjoldsvold

Andrew James Edwards

Wei Ting Chen

Sundar P Subramani

Jieqing Wang

Michael E Roberson

Krisjan David Fritz

Jianhua Yan

Michael Hauss

Kaviyarasan Rajendran

Ziehen Sun

Jun Chen

Wei Chen

A distributed storage system includes table controllers collectively storing partitions of a logical table having multiple rows. For each of the partitions, a corresponding table controller stores a subset of the rows of the logical table across individual pages. A management controller updates a schema of the logical table from a first to a second schema definition by, for each of the partitions, sending an update command indicating the second schema definition to the corresponding table controller. The first table controller corresponds to a first partition that includes a first subset of the rows. The first table controller, prior to receiving the update command, includes an identifier of the first schema definition within each page of data for the first subset. Subsequent to receiving the update command, the first table controller, when writing each new page of data for the first subset, includes an identifier of the second schema definition.

Online schema change of range-partitioned index in a distributed storage system

Anchor tree cross-merge within a distributed storage system. A computer system identifies a data structure that includes (i) a root anchor tree and (ii) an ordered set of anchor trees that are ordered based on their creation, such that a last anchor tree of the ordered set of anchor trees is most-recently created. The data structure stores index data for a set of objects stored in a non-volatile storage. The computer system creates a new anchor tree in the ordered set of anchor trees. The computer system identifies, from a set of delta tables, index data representing one or more objects that are stored on the non-volatile storage. The computer system merges the index data representing the one or more objects into the new anchor tree.

Staging anchor trees for improved concurrency and performance in page range index management

A distributed storage system includes non-volatile storage storing portions of a first object. The first object encompasses data having a first range of addresses and each portion includes data for a respective range of addresses that is a proper subset of the first range. A first data structure stores, for each portion, data indicating the respective range of addresses and a pointer to where the portion is stored. The first data structure includes a root tree and a set of trees ordered by creation data such that a last tree is most-recently created. The non-volatile storage stores received write data and a write buffer stores index data pointing to storage locations of the received write data. An index management system stores the index data from the write buffer into the last tree and, if the ordered set is empty, creates a tree in the ordered set before the storing.

A service enables a command that refers to a file system object using a hierarchical namespace identifier to be executed against the file system object in a flat namespace. The service selectively distributes the command to one of a plurality of name resolution nodes based on a directory name included in the hierarchical namespace identifier. The identified node resolves the directory name to a flat namespace identifier that is used to execute the command against the flat namespace. After communicating with at least one storage node to resolve a directory name, each name resolution node stores a mapping of the directory name to the corresponding flat namespace identifier in a cache, so that subsequent resolutions of that directory name may be performed more efficiently. Cache entries may be invalidated when an operation occurs that impacts the relevant mapping and/or based on system considerations such as cache expiry.

Hierarchical namespace service with distributed name resolution caching and synchronization

Embodiments provide a method to collect aggregate information or usage data quickly and efficiently with minimal lag. Additionally, the system can use this aggregate information internally for improved load balancing, better data placement, optimization, and enhanced debugging. The system can quickly look at aggregate information across a huge amount of data and drill down cheaply because the aggregate information is generated using existing processes. Aggregated statistics storage and collection may be built on top of an LSM tree used to store a persistent index for a cloud storage system. The statistics may also represent the result of an operation (e.g., max, min, sum, average) on selected parameter(s) or attribute(s) of stored data. Aggregate statistics values may be efficiently maintained during index merge and garbage collection processes or any other index management. As delta LSM trees are merged into a base LSM tree, the aggregates are updated in delta fashion.

File table index aggregate statistics

A cloud storage system includes a processor and a non-transitory computer-readable medium to store blob table management instructions for execution by the processor. The blob table management instructions are configured to manage a plurality of storage requests for a blob stored in a storage stamp as snapshots in a blob table and selectively create a user snapshot of at least one of the snapshots in the blob table. When automatic snapshots are enabled, the blob table management instructions are configured to receive a first request to overwrite the blob. If the first request does not further specify a key of the one of the snapshots in the blob table, the blob table management instructions are configured to add a new snapshot to the blob table and maintain storage of a prior snapshot of the blob for a maximum period.

Distributed data storage system with automatic snapshots, user snapshots and soft delete

Methods, systems, and programs provide for executing distributed transactions in a cloud storage system with a hierarchical namespace. One method includes receiving a request with operations to be executed atomically. Further, nodes are identified for executing the operations, each node having a respective clock and having at least part of a transactions table for controlling updates to entities. Each clock is one of a loosely-synchronized, a strictly-synchronized clock, a logical, or a physical clock. Additionally, the nodes process the operations, which includes setting a commit timestamp (CS) to a value of the clock in the node if the node is a first node in the processing. One node coordinates the transactions, and may be one of the nodes executing transactions. If the clock in the node is less than a current value of the CS, the node waits for the clock to reach the current value of the CS and the CS is updated. The transactions table is updated based on the value of the CS, the atomic execution is committed utilizing the final value of the CS, and a status is returned.

Distributed transactions in cloud storage with hierarchical namespace

Methods, systems, and apparatuses are provided for a storage system that implements a hierarchical namespace service. A storage system includes a plurality of physical nodes and a plurality of sets of virtual nodes. Each set of virtual nodes is managed by a corresponding physical node. Each virtual node is configured to manage a respective set of directory blocks. Each directory block is a respective partition of a storage namespace and is managed by a corresponding single virtual node. Each virtual node maintains a directory block map. The directory block map maps file system object names in a hierarchical namespace to entity block identifiers in the flat namespace for entity blocks (files and folders) stored in directories corresponding to the managed set of directory blocks. Load balancing may be performed by moving virtual nodes between physical nodes, and by splitting directory blocks.

Hierarchical namespace with strong consistency and horizontal scalability

Methods for dequeue optimizations in queues are performed by systems and apparatuses. The methods optimize dequeue operations using aggregation of expired messages enqueued in a queue and conditional iteration over enqueued messages based on the aggregation to service dequeue commands. Queues utilize page hierarchies such as root pages, index pages, and data pages. The aggregation of expired messages for pages in the queue determines the latest expired time for messages for a given page, and these latest expired times are stored in their respective pages, including data pages, index pages, and root pages, for use in the conditional iteration. The conditional iteration bypasses pages for which a latest expired time for all messages is prior to the current time when servicing dequeue requests for the queue.

System and method for dequeue optimization using conditional iteration

Growing community of inventors

Kenmore, WA, United States of America

Quan Zhang