Microsoft Technology Licensing, LLC

Lu Xun

Gopala Krishna R Kakivaya

Richard L Hasha

Gopala Krishna Reddy Kakivaya

Muralidhar Krishnaprasad

Dahlia Malkhi

Rishi Rakesh Sinha

Ajay Kalhan

Santeri Olavi Voutilainen

Radhakrishnan Srikanth

Yang Li

Hua-Jun Zeng

Mihail Gavril Tarta

Anurag Gupta

Dharma Kiritkumar Shukla

Lidong Zhou

Alex Wun

Muthian Sivathanu

Rimma V Nehme

Gopal Kakivaya

Vipul A Modi

Ankur Agrawal

Mansoor Mohsin

Balachandar Pavadaisamy

Anil Kumar Nori

Jason Todd Hunter

Pehkeong Teh

Manoj K Valiyaparambil

Matthew Snider

Gopala Kirhsna R Kakivaya

The disclosure herein describes platform-level migration for deep learning training (DLT) jobs from a checkpointed stated between a source node and a destination node. The checkpointing is performed through capturing GPU state (e.g., device state) and CPU state (e.g., host state). The GPU state includes GPU data (e.g., model parameters, optimizer state, etc.) that is located in the GPU and GPU context (e.g., the default stream in GPU, various handles created by libraries). Restoring the DLT job on the destination node involves resumption of processing of a destination GPU at the same checkpointed state.

Artificial intelligence workload migration for planet-scale artificial intelligence infrastructure service

Systems and methods that restore a failed reconfiguration of nodes in a distributed environment. By analyzing reports from read/write quorums of nodes associated with a configuration, automatic recovery for data partitions can be facilitated. Moreover, a configuration manager component tracks current configurations for replication units and determines whether a reconfiguration is required (e.g., due to node failures, node recovery, replica additions/deletions, replica moves, or replica role changes, and the like.) Reconfigurations of data activated as being replicated from an old configuration to being replicated on a new configuration may be performed in a transactionally consistent manner based on dynamic quorums associated with the new configuration and the old configuration.

Quorum based transactionally consistent membership management in distributed storage

Techniques are described herein for storing instances of data among nodes of a distributed store based on hierarchical scale unit values. Hierarchical scale unit values are assigned to the respective nodes of the distributed store. A first instance (e.g., a primary instance) of a data module is stored in a first node having a first hierarchical scale unit value. A primary instance of the data module with respect to a data operation is an instance of the data module at which the data operation with respect to the data module is initiated or initially directed. A second instance (e.g., a primary or secondary instance) of the data module is stored in a second node having a second hierarchical scale unit value based on a magnitude of a difference between the first hierarchical scale unit value and the second hierarchical scale unit value. A secondary instance is essentially a 'back-up' instance.

Hierarchical scale unit values for storing instances of data

In various implementations, methods and systems resource balancing in a distributed computing environment are provided. A client defined resource metric is received that represents a resource of nodes of the cloud computing platform. A placement plan for job instances of service applications is generated. The placement plan includes one or more movements that are executable to achieve a target placement of the job instances on the nodes. It is determined that the placement plan complies with placement rules. Each placement rule dictates whether a given job instance of the job instances is suitable for placement on a given node of the nodes. The placement plan is executed based on determining that the target placement of the job instances improves balance of resources across the nodes of the cloud computing platform based on the resource represented by the client defined resource metric.

Balancing resources in distributed computing environments

Examples of the disclosure provide for providing a global timestamp in a distributed system. A plurality of nodes are connected via a network. A logical clock is implemented on a node of the plurality of nodes. The logical clock sets a global time for the network, such that another node of the plurality of nodes labels a local event with a global timestamp interval based on the global time provided by the logical clock.

Global logical timestamp

Systems and methods that restore failed reconfiguration of nodes in distributed systems. By analyzing reports from read/write quorums of nodes associated with a configuration, automatic recovery for data partitions can be facilitated. Moreover, a configuration manager component tracks current configurations for replication units and determines whether a reconfiguration is to be performed (e.g., due to node failures, node recovery, replica additions/deletions, replica moves, or replica role changes, and the like.) Reconfigurations of data activated as being replicated from a first configuration to a second configuration may be performed in a transactionally consistent manner based on dynamic quorums associated with the second configuration and the first configuration.

Quorum based transactionally consistent membership management in distributed systems

Embodiments are directed to a cluster of nodes configured to run one or more services. A gateway on each node is configured to provide communication between the node and clients. The gateways manage filters designated by the clients, wherein the filters are named based upon a naming format used to name the services. The gateways receive configuration updates broadcast by a failover manager. The configuration updates comprise service endpoint changes for services running on the nodes. The gateways provide notifications to clients when a filter name matches at least a portion of a service name in a configuration update. The configuration updates identify a configuration version, and the gateways may provide notifications with endpoint information from more than one configuration update version. Notifications may be configured to be sent only when the endpoint for a primary replica of a service has changed.

Scalable event-based notifications

Architecture that provides reliable communications of broadcast data (e.g., a message) in a collection of nodes. Each node in the collection is assigned a range of identifiers in a token. The union of the tokens for all nodes is the entire identifier range space. Each node that receives a reliable broadcast message from an originator node acknowledges receipt of the message using its token. One or more intermediate nodes forward the message from the originator node to other nodes with which the originator node has no direct communications (multi-level node structure). The indirect nodes each send acknowledgements back to the parent nodes (which can be an intermediate node) which combine the tokens to ensure the entire range space for the associated assigned token range is covered. The originator node ultimately receives tokens to compute if all nodes have received the message.

Reliable broadcast in a federation of nodes

Determining if a new federation should be formed. A method includes a voter node determining that it has sufficient votes to form a new federation. The method further includes determining that no node existing in a previous federation already has a quorum of valid global tickets. Each global ticket indicates that a node agrees that a federation should continue to exist. As a result, the voter node forms a new federation.

Cluster voter model

Techniques are described herein that are capable of establishing causality order of computer trace records. A computer trace record is information that indicates an event that occurs with regard to execution of a computer program. For instance, machines in a distributed computer system may generate such computer trace records as the machines execute the computer program. The computer trace records may be ordered in an aggregated trace to accurately reflect the causality order of the computer trace records. A causality order of computer trace records is a temporal sequence of the computer trace records in which each cause event is indicated to occur before each effect event that is caused by the cause event. A cause event is an event that causes an effect event. An effect event is an event that is caused by a cause event.

Establishing causality order of computer trace records

The present invention extends to methods, systems, and computer program products for a joining node to join a ring of nodes within a rendezvous federation. Embodiments of the invention include establishing a neighborhood of multiple nodes on the ring of nodes. The joining node indicates its intent to take id-space ownership for a portion of the id-space between the joining node and a selected immediately adjacent node. The joining node initiates a one-way monitoring relationship with the selected immediately adjacent node. The joining node receiving an indication from the selected immediately adjacent node that indicates acceptance of the joining node's intent to take id-space ownership for a portion of the id-space between the joining node and the selected node and indicates establishment of a one-way monitoring relationship between the selected node and the joining node. The joining node agrees to participate in a one-way monitoring relationship with the selected node.

Maintaining consistency within a federation infrastructure

Obtaining tight knowledge about nodes in a federated ring structure. The tight knowledge includes information about neighbors of a given node. The method includes at a given node, sending a join request, requesting to join a federated ring in a distributed computing environment. The method further includes, at the given node, receiving a join reply. The join reply provides an initial view of at least a portion of a neighborhood of the given node. The neighborhood is a set of nodes logically proximate the given node. The method further includes, at the given node, sending a lock request to nodes in the neighborhood to request that the other nodes in the given node's neighborhood are locked so as to not have overlapping nodes joining the given node's neighborhood simultaneously.

Growing community of inventors

Kirkland, WA, United States of America

Lu Xun