International Business Machines Corporation

Abdullah Kayi

Alper Buyuktosunoglu

Wei Zhang

Xiaodong Cui

Charles Ray Johns

James Allan Kahle

Constantinos Evangelinos

Yoonho Park

Jose R Brunheroto

Carlos Henrique Andrade Costa

Changhoan Kim

Youssef Mroueh

Patrick D Siegl

Mingrui Liu

Ravi Nair

Martin Ohmacht

Tayfun Gokmen

James C Sexton

Alessandro Morari

A system comprises compute nodes distributed over a network and configured to perform a pipeline parallel process. The system also comprises an extended memory comprising a global virtual address space which is shared by the compute nodes. The extended memory is configured to enable the compute nodes to exchange data over the network when the compute nodes perform the pipeline parallel process.

Pipeline parallel computing using extended memory

Embodiments of a method are disclosed. The method includes performing distributed deep learning training on multiple batches of training data using corresponding learners. Additionally, the method includes determining training times wherein the learners perform the distributed deep learning training on the batches of training data. The method also includes modifying a processing aspect of the straggler to reduce a future training time of the straggler for performing the distributed deep learning training on a new batch of training data in response to identifying a straggler of the learners by a centralized control.

Dynamic computation rates for distributed deep learning

Embodiments of a method are disclosed. The method includes performing distributed deep learning training on a batch of training data. The method also includes determining training times representing an amount of time between a beginning batch time and an end batch time. Further, the method includes modifying a communication aspect of the communication straggler to reduce a future network communication time for the communication straggler to send a future result of the distributed deep learning training on a new batch of training data in response to the centralized parameter server determining that the learner is the communication straggler.

Dynamic network bandwidth in distributed deep learning training

Embodiments of a method are disclosed. The method includes performing decentralized distributed deep learning training on a batch of training data. Additionally, the method includes determining a training time wherein the learner performs the decentralized distributed deep learning training on the batch of training data. Further, the method includes generating a table having the training time and other processing times for corresponding other learners performing the decentralized distributed deep learning training on corresponding other batches of other training data. The method also includes determining that the learner is a straggler based on the table and a threshold for the training time. Additionally, the method includes modifying a processing aspect of the straggler to reduce a future training time of the straggler for performing the decentralized distributed deep learning training on a new batch of training data in response to determining the learner is the straggler.

Dynamic computation in decentralized distributed deep learning training

Systems, computer-implemented methods, and computer program products to facilitate updating, such as averaging and/or training, of one or more statistical sets are provided. According to an embodiment, a system can comprise a memory that stores computer executable components and a processor that executes the computer executable components stored in the memory. The computer executable components can include a computing component that averages a statistical set, provided by the system, with an additional statistical set, that is compatible with the statistical set, to compute an averaged statistical set, where the additional statistical set is obtained from a selected additional system of a plurality of additional systems. The computer executable components also can include a selecting component that selects the selected additional system according to a randomization pattern.

Updating of statistical sets for decentralized distributed training of a machine learning model

Embodiments of a method are disclosed. The method includes performing a batch of decentralized deep learning training for a machine learning model in coordination with multiple local homogenous learners on a deep learning training compute node, and in coordination with multiple super learners on corresponding deep learning training compute nodes. The method also includes exchanging communications with the super learners in accordance with an asynchronous decentralized parallel stochastic gradient descent (ADPSGD) protocol. The communications are associated with the batch of deep learning training.

Hierarchical decentralized distributed deep learning training

An approach is disclosed that configures a computer system node from components that are each connected to an intra-node network. The configuring is performed by selecting a set of components, including at least one processor, and assigning each of the components a different address range within the node. An operating system is run on the processor included in the node with the operating system accessing each of the assigned components.

Disaggregated system domain

An approach is disclosed that maintains a consistent view of a virtual address by a local node which writes a first value to the virtual address and, after writing the first value, establishes a snapshot consistency state of the virtual address. The virtual address is shared amongst any number of processes and the processes includes a writing process and other processes that read from the virtual address. After writing the first value, the writing process writes a second value to the virtual address. Even after writing the second value, the first value is still visible to the other processes.

Global virtual address space consistency model

Methods and systems for shuffling data are described. A processor may generate pair data from source data. The processor may insert the pair data into local tuple spaces. In response to a request for a particular key, the processor may determine a presence of the requested key in a global tuple space. The processor may, in response to a presence of the requested key in the global tuple space, update the global tuple space. The update may be based on the pair data among the local tuple spaces including the existing key. The processor may, in response to an absence of the requested key in the global tuple space, insert pair data including the missing key from the local tuple spaces into the global tuple space. The processor may fetch the requested pair data, and may shuffle the fetched data to generate a dataset.

Data shuffling with hierarchical tuple spaces

An approach is described that accesses data in a shared memory that is shared amongst nodes that include a local node and remote nodes. The local node receives a name corresponding to a named data element in a Coordination Namespace, the Coordination Namespace having been created in a memory distributed amongst the nodes. A hash function is applied to at least a portion of the name with a result of the hash function being a natural node indicator. Data corresponding to the named data element is requested from a natural node identified by the indicator. Based on the request, a response is received from the natural node.

Coordination namespace processing

Methods and systems for shuffling data to generate a dataset are described. A first map module may generate first pair data, and a second map module may generate second pair data, from source data. The first map module may insert the first pair data into a first local tuple space accessible to the first map module. The second map module may insert the second pair data into a second local tuple space accessible to the second map module. A shuffle module may request pair data that includes a particular key. The first and second pair data may be inserted into a global tuple space accessible by the first and second map modules. The shuffle module may identify the requested pair data in the global tuple space, and may fetch the identified pair data from a memory. The shuffle module may shuffle the fetched pair data to generate the dataset.

Growing community of inventors

West Chester, NY, United States of America

Abdullah Kayi