Microsoft Technology Licensing, LLC

Arild Einar Skjolsvold

Bradley Gene Calder

Ju Wang

Hemal Khatri

Shane Kumar Mainali

Shashwat Srivastav

Niranjan Nilakantan

Xinhua Ji

Jiesheng Wu

Padmanabha Chakravarthy Uddaraju

Abdul Rafay Abbasi

Maneesh Sah

Jegan Devaraju

Rushi Srinivas Surla

Harshawardhan Gadgil

Sriram Sankaran

Yang Yang Zhang

Maxim Mazeev

Pradeep Kumar Gunda

Kavitha Manivannan

Marvin McNett, Ii

Shyam Antony

Vaman Bedekar

Leonidas Rigas

Ralph Burton Harris, Iii

Hosung Song

Samuel James McKelvie

Aaron William Ogus

Wei Lin

Marcus Swenson

Deepali Bhardwaj

Yongfu Lou

Girish Saini

Hakon Gabriel Verespej

Qiuhan Wang

Donglin Wei

Sumadhur Reddy Bolli

Liang Xie

Dengkui Xi

Dengyao Mo

Yang Zhang

Marcus Kimball Swenson

Marvin Mcnett Ii

Ralph Burton Harris Iii

Systems and methods are provided that enable a general framework for partitioning application-defined jobs in a scalable environment. The general framework decouples partitioning of a job from the other aspects of the job. As a result, the effort required to define the application-defined job is reduced or minimized, as the user is not required to provide a partitioning algorithm. The general framework also facilitates management of masters and servers performing computations within the distributed environment.

Managing partitions in a scalable environment

Techniques are provided for managing resources among clusters of computing devices in a computing system. Resource reassignment message are generated for indicating that servers are reassigned and in response to resource compute loads exceed or fall below certain thresholds. Techniques also include establishing communications with the reassigned servers to assign compute loads without physically relocating the servers from one cluster to another cluster.

Cluster resource management in distributed computing systems

Presented herein are methods of replicating versioned and hierarchical data structures, as well as data structures representing complex transactions. Due to interdependencies between data entities and a lack of guaranteed message ordering, simple replication methods employed for simple data types cannot be used. Operations on data structures exhibit dependencies between the messages making up the operations. This strategy can be extended to various types of complex transactions by considering certain messages to depend on other messages or on the existence of other entries at the data store. Regardless of origin, these dependencies can be enforced by suspending the processing of messages with unsatisfied dependencies until all of its dependencies have been met. Alternately, transactions can be committed immediately, creating entities that include versioned identifiers for each of their dependencies. These entities can then be garbage collected of the parent objects are not subsequently created.

Versioned and hierarchical data structures and distributed transactions

Techniques for managing resources among clusters of computing devices in a computing system are described herein. In one embodiment, a method includes receiving, via a computer network, a resource reassignment message indicating that a server is reassigned from a first cluster to a second cluster and in response to the received resource reassignment message, establishing communications with the server reassigned from the first cluster to the second cluster via the computer network. The method further includes subsequent to establishing communications with the server via the computer network, assigning a compute load to the server reassigned from the first cluster to the second cluster without physically relocating the server from the first cluster to the second cluster.

One example provides a method of migrating a data partition from a first storage cluster to a second storage cluster, the method including determining that the data partition meets a migration criteria for migrating from the first storage cluster to the second storage cluster, on the first storage cluster, preparing partition metadata to be transferred, the partition metadata describing one or more streams within the data partition and one or more extents within each stream, transferring the partition metadata from the first storage cluster to the second storage cluster, directing new transactions associated with the data partition to the second storage cluster, including while the one or more extents reside at the first storage cluster, on the first storage cluster, changing an access attribute of the one or more extents within the data partition to read-only, and on the second storage cluster, performing new ingress for the data partition.

Data partition migration via metadata transfer and access attribute change

Live migration of partitions can be achieved using server-to-server communications to coordinate and control partition migration and rebuild a persistent state of the partition on a destination server while the partition is served on the source server. The live migration approach breaks the heavy dependency on the partition master by eliminating most of the server-to-master communications, delegating coordination and control of the migration to the source and destination servers. After the migration is initiated by the partition master, the source server sends in-memory cache entries indicating the persistent state of the partition to the destination server. Once transferred, the two servers can cooperate to complete the handover, which loads the partition on the destination server using the cache entries received from the source server. Once the source server offloads the partition, it can redirect traffic for the partition to the destination server by triggering an update to the partition map.

Live migration of partitions

Systems and methods are provided that enable a general framework for partitioning application-defined computations (e.g., jobs) in a scalable environment. The general framework decouples partitioning of a computation from the other aspects of the computation. As a result, the effort required to define an application-defined job is reduced or minimized, as the user is not required to provide a partitioning algorithm. A user can optionally take advantage of a partitioning framework by providing application-defined interfaces to perform the desired job. Optionally, a user can provide additional information to allow for modification of how partitions are assigned.

Decoupling partitioning for scalability

A system for load-balancing partitions of file servers of a file storage service of a cloud computing system is provided. The system collects latency information relating to access requests to access storage that is partitioned. The access requests are processed by the file servers. Each file server is assigned partitions, which are indexes to the storage. For each file server, the system generates, from the latency information, partition statistics relating to partition latency associated with each partition assigned to that file server. For each partition, the system applies rules to the statistics to determine actions to take to mitigate performance problems. Each rule has a condition and an action. When the condition of the rule is satisfied based on the partition statistics for the partition, the system records an indication to take the action of the rule.

Tracking internal latencies for load balancing of file server

A traffic manager for a distributed data storage system includes an iterative spike identifier to identify N levels of traffic spikes in traffic data on an account basis in a distributed data storage system, where N is greater than zero. A traffic cycle identifier selectively identifies cyclic traffic spikes in at least one of the N levels of traffic spikes for each of a plurality of accounts using autocorrelation and peak detection. A partition manager communicates with the traffic cycle identifier and selectively partitions one of the plurality of accounts based on based on the autocorrelation and the peak detection.

Volatile account identification and isolation and resource management in distributed data storage systems

Partition management for a scalable, structured storage system is provided. The storage system provides storage represented by one or more tables, each of which includes rows that represent data entities. A table is partitioned into a number of partitions, each partition including a contiguous range of rows. The partitions are served by table servers and managed by a table master. Load distribution information for the table servers and partitions is tracked, and the table master determines to split and/or merge partitions based on the load distribution information.

Partition management in a partitioned, scalable, and available structured storage

Growing community of inventors

Kenmore, WA, United States of America

Arild Einar Skjolsvold