Microsoft Technology Licensing, LLC

Other

Shane Kumar Mainali

Maneesh Sah

Ju Wang

Bradley Gene Calder

Andrew James Edwards

Krishnan Varadarajan

Shashwat Srivastav

Jiesheng Wu

Niranjan Nilakantan

Padmanabha Chakravarthy Uddaraju

Abdul Rafay Abbasi

Jegan Devaraju

Quan Zhang

Zichen Sun

Rushi Srinivas Surla

Arild Einar Skjolsvold

Venkates Paramasivam Balakrishnan

Shaoyu Zhang

Hemal Khatri

Manish Chablani

Georgi M Chalakov

Jun Tse Chen

Thomas Leo Marquardt

Hao Feng

Esfandiar Manii

Saurabh Pant

Dana Yulian Kaban

Saher B Ahwal

Da Zhou

Amit Pratap Singh

Junhua Gu

Aaron William Ogus

Yikang Xu

Jingchao Zhang

Weiping Zhang

Wei Zhu Chen

Cheng Huang

Ivan Brugiolo

Huseyin Simitci

Ioan Oltean

Ovais Khan

Harshawardhan Gadgil

Manan Shah

Sivakumar Kalva

Pradeep Seela

Sridhar Srinivasan

Ishai Menache

Peter Bodik

Wei Ting Chen

Maxim Mazeev

He Su

Leonidas Rigas

Yongfu Lou

Michael E Roberson

Jianwu Gao

Kaifeng Zhu

Bin Tong

Pei Zhang

Yi-Fan Tsai

Kristopher T Lange

Arild Einar Skjoldsvold

Yang Lu

Sandesh Doddameti

Wei Lin

Sundar P Subramani

Muhammad Junaid Shahid

Jieqing Wang

James Lee Baker

Krisjan David Fritz

Girish Saini

Hao Wang

Vamshi Konagari

Ramy Wassef

Jun Chen

Wei Chen

Advait Kumar Mishra

Ziehen Sun

Kaviyarasan Rajendran

Michael Hauss

Jianhua Yan

Jegan Devanraju

Embodiments of the present invention relate to synchronously replicating data in a distributed computing environment. To achieve synchronous replication both an eventual consistency approach and a strong consistency approach are contemplated. Received data may be written to a log of a primary data store for eventual committal. The data may then be annotated with a record, such as a unique identifier, which facilitates the replay of the data at a secondary data store. Upon receiving an acknowledgment that the secondary data store has written the data to a log, the primary data store may commit the data and communicate an acknowledgment of success back to the client. In a strong consistency approach, the primary data store may wait to send an acknowledgement of success to the client until it receives an acknowledgment that the secondary has not only written, but also committed, the data.

Synchronous replication in a distributed storage environment

A distributed storage system includes table controllers collectively storing partitions of a logical table having multiple rows. For each of the partitions, a corresponding table controller stores a subset of the rows of the logical table across individual pages. A management controller updates a schema of the logical table from a first to a second schema definition by, for each of the partitions, sending an update command indicating the second schema definition to the corresponding table controller. The first table controller corresponds to a first partition that includes a first subset of the rows. The first table controller, prior to receiving the update command, includes an identifier of the first schema definition within each page of data for the first subset. Subsequent to receiving the update command, the first table controller, when writing each new page of data for the first subset, includes an identifier of the second schema definition.

Online schema change of range-partitioned index in a distributed storage system

Methods, systems, apparatuses, and computer program products are provided that enable immediate access to data associated with a source blob. The foregoing is achieved by creating a copy-on-read blob based on the source blob and a link blob based on the copy-on-read blob for each virtual machine to be deployed. When creating the copy-on-read blob, the source blob is not copied. Instead, data from the source blob is copied to the copy-on-read blob upon request by a virtual machine. Subsequent requests for that data are retrieved from the copy-on-read blob. Each link blob stores write data provided by its corresponding virtual machine. When a virtual machine requests data, its corresponding link blob is checked for the data. If the data is not stored therein, the copy-on-read blob is checked for the data. If the copy-on-read blob does not store the data, the data is retrieved from the source blob.

Providing instant and distributed access to a source blob via copy-on-read blobs and link blobs

Anchor tree cross-merge within a distributed storage system. A computer system identifies a data structure that includes (i) a root anchor tree and (ii) an ordered set of anchor trees that are ordered based on their creation, such that a last anchor tree of the ordered set of anchor trees is most-recently created. The data structure stores index data for a set of objects stored in a non-volatile storage. The computer system creates a new anchor tree in the ordered set of anchor trees. The computer system identifies, from a set of delta tables, index data representing one or more objects that are stored on the non-volatile storage. The computer system merges the index data representing the one or more objects into the new anchor tree.

Staging anchor trees for improved concurrency and performance in page range index management

Example aspects include techniques for ordering operations in a file system having a hierarchical namespace including determining, by a first service that manages file system operations for a first entity in the hierarchical namespace, to perform a file system operation for the first entity, generating, by the first service, a first counter record for the file system operation, obtaining, based on determining to perform the file system operation and from a second service that manages file system operations on a second entity that is a parent of the entity in the hierarchical namespace, a second counter record for a last file system operation performed on the second entity, wherein the second counter record is generated by the second service, and storing, in a change feed, an indication of the file system operation along with the first counter record and the second counter record.

Methods and systems for ordering operations on a file system having a hierarchical namespace

Presented herein are methods of replicating versioned and hierarchical data structures, as well as data structures representing complex transactions. Due to interdependencies between data entities and a lack of guaranteed message ordering, simple replication methods employed for simple data types cannot be used. Operations on data structures exhibit dependencies between the messages making up the operations. This strategy can be extended to various types of complex transactions by considering certain messages to depend on other messages or on the existence of other entries at the data store. Regardless of origin, these dependencies can be enforced by suspending the processing of messages with unsatisfied dependencies until all of its dependencies have been met. Alternately, transactions can be committed immediately, creating entities that include versioned identifiers for each of their dependencies. These entities can then be garbage collected of the parent objects are not subsequently created.

Versioned and hierarchical data structures and distributed transactions

A distributed storage system includes non-volatile storage storing portions of a first object. The first object encompasses data having a first range of addresses and each portion includes data for a respective range of addresses that is a proper subset of the first range. A first data structure stores, for each portion, data indicating the respective range of addresses and a pointer to where the portion is stored. The first data structure includes a root tree and a set of trees ordered by creation data such that a last tree is most-recently created. The non-volatile storage stores received write data and a write buffer stores index data pointing to storage locations of the received write data. An index management system stores the index data from the write buffer into the last tree and, if the ordered set is empty, creates a tree in the ordered set before the storing.

In various embodiments, methods and systems for implementing a distributed metadata management system in distributed storage systems are provided. A distributed storage system operates based on data storage resources (e.g., extents and streams). The distributed metadata management system is implemented for extent and stream metadata to facilitate the scalability of metadata processing. The distributed storage system implements extent managers and stream managers that independently manage extent and stream metadata, respectively. The extent managers are associated with an extent table that stores extent metadata. The stream managers are associated with streams that store associations with extents. The distributed metadata management system can also utilize a bootstrap layer that leverages components of a legacy distributed storage system to facilitate distributed management of extent and stream metadata. The bootstrap layer is used to store the extent table as a system table and to persist the state of the stream manager as system streams.

Distributed metadata management in a distributed storage system

A service enables a command that refers to a file system object using a hierarchical namespace identifier to be executed against the file system object in a flat namespace. The service selectively distributes the command to one of a plurality of name resolution nodes based on a directory name included in the hierarchical namespace identifier. The identified node resolves the directory name to a flat namespace identifier that is used to execute the command against the flat namespace. After communicating with at least one storage node to resolve a directory name, each name resolution node stores a mapping of the directory name to the corresponding flat namespace identifier in a cache, so that subsequent resolutions of that directory name may be performed more efficiently. Cache entries may be invalidated when an operation occurs that impacts the relevant mapping and/or based on system considerations such as cache expiry.

Hierarchical namespace service with distributed name resolution caching and synchronization

Data is replicated between primary and secondary storage systems using a data pulling process or a log shipping process. If data is to be replicated through the log shipping process, a transaction and its associated data get stored in a transaction log in the secondary storage system. Subsequently, when the transaction gets executed, actual data gets persisted from the transaction log to a data log in the secondary storage system. If the data is to be replicated through the data pulling process, the transaction is stored in the transaction log. However, the associated data for the transaction is retrieved from the first primary storage system and stored directly in the data log in the secondary system.

Data replication in a distributed storage system

One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Data partition migration via metadata transfer and access attribute change

A snapshot of data from a table associated with a particular user may be generated. Tree data structures that are distributed across multiple computer systems may be accessed. Each of the tree structures may include data associated with one or more users. At least one tree data structure of the tree data structures that includes data associated with the particular user of the one or more users may be identified. The at least one tree data structure may then be filtered. Filtering may comprise identifying only data that is associated with the particular user. A snapshot of the data associated with the particular user may be generated. Generating the snapshot of the data associated with the particular user comprises generating a data structure that is configured to map to each data page of the at least one tree data structure that includes data associated with the particular user.

Efficient snapshot generation of data tables

Growing community of inventors

Duvall, WA, United States of America

Shane Kumar Mainali