Microsoft Technology Licensing, LLC

Bhalakumaaran Erode Ranganathan

Pankaj Sharma

Dharma Kiritkumar Shukla

Karthik Raman

Momin Mahmoud Al-Ghosien

Ji Huang

Shireesh Kumar Thota

Madhan Gajendran

Atul Katiyar

Ankur Savailal Shah

Mikhail Mikhailovich Koltachev

Shreshth Singhal

Sujit Vattathil Kuruvilla

Ovidiu Constantin Platon

Subramanyam Pattipaka

Digvijaysinh Govindbhai Makwana

Ramachandran Ramjee

Muthian Sivathanu

Rimma V Nehme

Nipun Kwatra

Lan Tang

Vaibhav Sharma

Suresh Parameshwar

Smriti Yamini

Srinidhi Viswanatha

Krishna Pradeep Reddy Tamma

Padma Priya Aradhyula Bhavani

Bhavya Malik

Rama Krishnan Raghupathy

Ailidani Ailijiang

Ling Yu

Harini Chintapally

The disclosure herein describes platform-level checkpointing for deep learning (DL) jobs. The checkpointing is performed through capturing two kinds of state data: (i) GPU state (device state), and (ii) CPU state (host state). The GPU state includes GPU data (e.g., model parameters, optimizer state, etc.) that is located in the GPU and GPU context (e.g., the default stream in GPU, various handles created by the libraries such as DNN, Blas, etc.). Only a fraction of the GPU memory is copied because the checkpointing is done in a domain-aware manner. The 'active' memory contains useful data like model parameters. To be able to capture the useful data, memory management is controlled to identify which parts of the memory are active. Also, to restore the destination GPU to the same context/state, a mechanism is used to capture such state-changing events on an original GPU and replayed on a destination GPU.

Transparent pre-emption and migration for planet-scale computer

A server set for a data set may designate a subset of 'master' servers that update the data set in order to reduce data version conflicts involving mutually exclusive updates of the data set. Multi-master configurations may fulfill the performance constraints, and the subset of masters may detect and resolve data version conflicts. However, if multiple masters perform conflict resolution for a particular data version conflict, the resolution may produce inefficiency and redundancy (if the masters reach the same outcome) or additional data version conflicts (if the masters reach different outcomes). Instead, among the masters, a merge master may be identified that applies conflict resolution techniques to data version conflicts and forwards the conflict resolution outcome to the other masters for application to the data set to resolve the data version conflict. The other masters may temporarily store updates in a tentative update set until data version conflicts are resolved.

Merging conflict resolution for multi-master distributed databases

Data services for workloads are often provided with a service level agreement specifying various performance guarantees (e.g., latency, availability, scalability, and consistency). Single-master architectures, in which updates to the data set are constrained to a single server, may limit the fulfillment of some performance guarantees. Presented herein are multi-master architectures, in which the server set is partitioned into at least two masters are permitted to update the data set and at least one non-master that is not permitted to update the data set. Non-masters that receive a request to update the data set forward the request to a master server for application to the data set. A master that receives the request applies it to the data set and propagates the update to other master and non-master servers. Conflicting updates may be resolved through a variety of conflict resolution techniques, optionally designating one master server as a conflict resolution server.

Multi-master architectures for distributed databases

Data services are often provided with various performance guarantees. Multi-master architectures, in which multiple master servers are permitted to update a portion of the data set, may facilitate some performance requirements, but may also lead to data version conflicts in which different masters have written different versions of one or more data items. Moreover, conflicts involving different data items may have to be resolved using different conflict resolution techniques. Therefore, various data items of the data set may be associated with a conflict resolution technique selected from a conflict resolution technique set, such as manual conflict resolution; a write order policy, such as last writer wins; a conflict resolution logic; and conflict resolution based on data types. A data version conflict may be resolved by identifying and invoking the selected conflict resolution technique that is associated with the data item, and applying the conflict resolution outcome to the data item.

Conflict resolution for multi-master distributed databases

Partitions are enabled to be merged in storage device of distributed database systems. A request to merge a plurality of source partitions into a merged partition is received. Each source partition stores a plurality of data items having corresponding source global identifiers and corresponding source local identifiers. The data items, with corresponding metadata, are appended from the source partitions to an intermediate storage structure in an order defined by the source local identifiers. Target global identifiers and target local identifiers are generated for each of the data items in the intermediate storage structure. The copied data items are flushed from the intermediate storage structure to the merged partition in an order defined by the target global identifiers. The source partitions and intermediate storage structure may then be purged of data, removed, and/or reused for further storage needs.

Distributed database systems partition merge

Embodiments disclosed herein are related to systems and methods for a scheduling manager or other element of a distributed computing system to schedule events performed by a various computing systems based on a priority of the events. The distributed computing system includes one or more processors and system memory having stored thereon executable instructions. When the executable instructions are executed by the processors, the computing system receives a new event that is to be added to a queue of events that are to be executed by the distributed computing system, determines that there is a current active event in the queue, compares a priority of the new event with a priority of the current active event, and based on the comparison, modifies an order of the events in the queue by updating the event in the queue that is to be the current active event.

Priority based scheduling

To coordinate distributed processing of the data files specified for the run by the set of analysis computers, the analysis computers and an analysis management computer are each configured to follow a convention regarding how the run is specified and performed. The convention includes the timing of when each run begins and ends, and the specification of the data files to be included in the run, and the identification of the analysis computers available for the run. The analysis management computer provides a mechanism through which the set of data files to be processed, and a state of processing of those files, can be communicated with available analysis computers. The analysis computers communicate their availability to participate in the run through the analysis management computer and receive information enabling each analysis computer to determine its own partitions of the set of data files to process.

Distributed data processing system

The automated validation of one or more test cases in a validation pass. The validation pass is enabled using a plurality of scenario runners, which are executable components that respond to instructions to run one or more scenarios of a validation pass by beginning to run through such scenarios during the validation pass. The validation pass may be associated with a maximum time such that upper limits on how long the scenario runners may run is set. This likewise limits the time it takes to validate the results of the scenario. The scenario runners provide events that occur during the running of their respective scenarios into an event collector. The events of the validation pass may then be evaluated against one or more test cases to determine whether or not the events have a state indicative of a success or failure of each test cases.

Validation using scenario runners

Growing community of inventors

Bellevue, WA, United States of America

Bhalakumaaran Erode Ranganathan