Microsoft Technology Licensing, LLC

Other

Niranjan Nilakantan

Bradley Gene Calder

Shashwat Srivastav

Padmanabha Chakravarthy Uddaraju

Shane Kumar Mainali

Jiesheng Wu

Abdul Rafay Abbasi

Ju Wang

Arild Einar Skjolsvold

Hemal Khatri

Deepali Bhardwaj

Shyamalan Pather

Holly Knight

Xinran Wu

Harshawardhan Gadgil

Robert H Gerber

Maxim Mazeev

Alexander Felsöbüki Nagy

Leonidas Rigas

Shahaf Abileah

Murali R Krishnan

Praveen Seshadri

Michael Anthony Faoro

Philip Lawrence Garrett

Charles Tete Mensa-Annan

Alexander Felsobuki Nagy

Phil Garrett

Darryn Owen Lavery

Embodiments of the present invention relate to synchronously replicating data in a distributed computing environment. To achieve synchronous replication both an eventual consistency approach and a strong consistency approach are contemplated. Received data may be written to a log of a primary data store for eventual committal. The data may then be annotated with a record, such as a unique identifier, which facilitates the replay of the data at a secondary data store. Upon receiving an acknowledgment that the secondary data store has written the data to a log, the primary data store may commit the data and communicate an acknowledgment of success back to the client. In a strong consistency approach, the primary data store may wait to send an acknowledgement of success to the client until it receives an acknowledgment that the secondary has not only written, but also committed, the data.

Synchronous replication in a distributed storage environment

Presented herein are methods of replicating versioned and hierarchical data structures, as well as data structures representing complex transactions. Due to interdependencies between data entities and a lack of guaranteed message ordering, simple replication methods employed for simple data types cannot be used. Operations on data structures exhibit dependencies between the messages making up the operations. This strategy can be extended to various types of complex transactions by considering certain messages to depend on other messages or on the existence of other entries at the data store. Regardless of origin, these dependencies can be enforced by suspending the processing of messages with unsatisfied dependencies until all of its dependencies have been met. Alternately, transactions can be committed immediately, creating entities that include versioned identifiers for each of their dependencies. These entities can then be garbage collected of the parent objects are not subsequently created.

Versioned and hierarchical data structures and distributed transactions

Partition management for a scalable, structured storage system is provided. The storage system provides storage represented by one or more tables, each of which includes rows that represent data entities. A table is partitioned into a number of partitions, each partition including a contiguous range of rows. The partitions are served by table servers and managed by a table master. Load distribution information for the table servers and partitions is tracked, and the table master determines to split and/or merge partitions based on the load distribution information.

Partition management in a partitioned, scalable, and available structured storage

Embodiments of the present invention relate to invoking and managing a failover of a storage account between partitions within a distributed computing environment, where each partition represents a key range of data for the storage account. The partitions affected by the failover include source partitions hosted on a primary storage stamp and destination partitions hosted on a secondary storage stamp, where the storage account's data is being actively replicated from the primary to the secondary storage stamp. Upon receiving a manual or automatic indication to perform the failover, configuring the source partitions to independently perform flush-send operations (e.g., distributing pending messages as a group) and then configuring the destination partitions to independently perform flush-replay operations (e.g., aggressively replaying currently pending transactions). Upon completing the flush-replay operations, designating the secondary storage stamp as a new primary storage stamp such that live traffic is directed to the new primary storage stamp.

Implementing failover processes between storage stamps

Embodiments of the present invention relate to asynchronously replicating data in a distributed computing environment. To achieve asynchronous replication, data received at a primary data store may be annotated with information, such as an identifier of the data. The annotated data may then be communicated to a secondary data store, which may then write the data and annotated information to one or more logs for eventual replay and committal at the secondary data store. The primary data store may communicate an acknowledgment of success in committing the data at the primary data store as well as of success in writing the data to the secondary data store. Additional embodiments may include committing the data at the secondary data store in response to receiving an instruction that authorizes committal of data through a identifier.

Asynchronous replication in a distributed storage environment

Embodiments of the present invention relate to systems, methods, and computer storage media for replicating data in a distributed computing environment utilizing a combination of replication methodologies. A full-object replication may be utilized to replicate a full state of an object from a primary data store to a secondary data store. A checkpoint created after initiating the full-object replication may be parsed to identify changes to the object that have been entered since initiating the full-object replication. This replication process is referred to as a delta-checkpoint replication methodology. Additionally, in an embodiment, a log-based replication methodology may be utilized. The log-based replication may communicate data from a log of the primary data store to the secondary data store. It is also contemplated in an exemplary embodiment that when the log-based replication fails to maintain a throughput threshold, one of the other replication methodologies may be initiated, at least temporarily.

Replication processes in a distributed storage environment

Embodiments of the present invention relate to invoking and managing load-balancing operation(s) applied to partitions within a distributed computing environment, where each partition represents a key range of data for a storage account. The partitions affected by the load-balancing operation(s) are source partitions hosted on a primary storage stamp and/or destination partitions hosted on a secondary storage stamp, where the primary and secondary storage stamps are located in geographically distinct areas and are equipped to replicate the storage account's data therebetween. The load-balancing operation(s) include splitting partitions into child partitions upon detecting an increased workload as a result of active replication, merging partitions to form parent partitions upon detecting a reduction in workload as a result of decreased processing-related resource consumption, or offloading partitions based on resource consumption. A service within a partition layer of the storage stamps is responsible for determining when to invoke these load-balancing operation(s).

Load balancing when replicating account data

Embodiments of the present invention relate to systems, methods and computer storage media for facilitating the structured storage of binary large objects (Blobs) to be accessed by an application program being executed by a computing device. Generally, the manipulation of Blobs in a structured storage system includes receiving a request for a Blob, which may be located by way of a Blob pointer. The Blob pointer allows for the data, such as properties, of the Blob to be identified and located. Expired properties are garbage collected as a manipulation of the Blob data within a structured storage system. In an embodiment, the Blob is identified by a key that is utilized within a primary structured index to located the requested Blob. In another embodiment, the requested Blob is located utilizing a secondary hash index. In an additional embodiment, the Blob is locate utilizing a file table.

Blob manipulation in an integrated structured storage system

A cloud computing platform contains a structured storage subsystem the provides scalable queues. The cloud computing platform monitors message throughput for the scalable queues and automatically increases or decreases subqueues that provide the operational functionality for each scalable queue. A visibility start time and cloud computing platform time are maintained for each message to provide an approximate first-in-first-out order for messages within each subqueue. A message in a subqueue may be available for processing when the current cloud computing time is greater than the visibility start of the message.

Scalable queues on a scalable structured storage system

Presented herein are methods of continuously verifying data and repairing errors introduced during replication. In a particular embodiment, a primary data store sends out information sufficient to create a checkpoint together with a checksum for the data being verified at that checkpoint. At the secondary data store, a checkpoint is created in accordance with the checkpointing information, and a checksum is calculated over the indicated data at the created checkpoint. If the calculated checksum disagrees with the received checksum, additional checksums are calculated over subranges of the indicated data and compared with corresponding checksums over the data at the primary data store. The checksums at the primary data store may be requested from the primary data store or calculated locally based on the received overall checksum. Once an erroneous entry is identified, it can then be re-replicated from the primary data store to restore data consistency.

Geo-verification and repair

Growing community of inventors

Redmond, WA, United States of America

Niranjan Nilakantan