International Business Machines Corporation

Yandong Wang

Li Feng Zhang

Yufei Ren

Wei Zhang

Michel H T Hack

Jian Tan

Xiaoqiao Meng

Michel Hack

Minwei Feng

Min Li

Xingbo Wu

Valentina Salapura

Wei Tan

Zhenhua Hu

Shicong Meng

Luna Xu

Li Zhang

A machine receives a first set of global parameters from a global parameter server. Multiple learner processors in the machine execute an algorithm that models an entity type using the first set of global parameters and a mini-batch of data known to describe the entity type. The machine generates a consolidated set of gradients that describes a direction for the first set of global parameters in order to improve an accuracy of the algorithm in modeling the entity type when using the first set of global parameters and the mini-batch of data. The machine transmits the consolidated set of gradients to the global parameter server. The machine then receives a second set of global parameters from the global parameter server, where the second set of global parameters is a modification of the first set of global parameters based on the consolidated set of gradients.

Parameter data sharing for multi-learner training of machine learning applications

A machine receives a first set of global parameters from a global parameter server. The first set of global parameters includes data that weights one or more operands used in an algorithm that models an entity type. Multiple learner processors in the machine execute the algorithm using the first set of global parameters and a mini-batch of data known to describe the entity type. The machine generates a consolidated set of gradients that describes a direction for the first set of global parameters in order to improve an accuracy of the algorithm in modeling the entity type when using the first set of global parameters and the mini-batch of data. The machine transmits the consolidated set of gradients to the global parameter server. The machine then receives a second set of global parameters from the global parameter server, where the second set of global parameters is a modification of the first set of global parameters based on the consolidated set of gradients.

Version vector-based rules are used to facilitate asynchronous execution of machine learning algorithms. The method uses version vector based rule to generate aggregated parameters and determine when to return the parameters. The method also includes coordinating the versions of aggregated parameter sets among all the parameter servers. This allows to broadcast to enforce the version consistency; generate parameter sets in an on-demand manner to facilitate version control. Furthermore the method includes enhancing the version consistency at the learner's side and resolving the inconsistent version when mismatching versions are detected.

Coordinated version control system, method, and recording medium for parameter sensitive applications

Various embodiments manage dynamic memory allocation data. In one embodiment, a set of memory allocation metadata is extracted from a memory heap space. Process dependent information and process independent information is identified from the set of memory allocation metadata based on the set of memory allocation metadata being extracted. The process dependent information and the process independent information at least identify a set of virtual memory addresses available in the memory heap space and a set of virtual memory addresses allocated to a process associated with the memory heap space. A set of allocation data associated with the memory heap space is stored in a persistent storage based on the process dependent information and the process independent information having been identified. The set of allocation data includes the process independent allocation information and a starting address associated with the memory heap space.

Restorable memory allocator

A storage controller of a machine receives training data associated with a neural network model. The neural network model includes a plurality of layers, and the machine further including at least one graphics processing unit. The storage controller trains at least one layer of the plurality of layers of the neural network model using the training data to generate processed training data. A size of the processed data is less than a size of the training data. Training of the at least one layer includes adjusting one or more weights of the at least one layer using the training data. The storage controller sends the processed training data to at least one graphics processing unit of the machine. The at least one graphics processing unit is configured to store the processed training data and train one or more remaining layers of the plurality of layers using the processed training data.

Storage controller acceleration for neural network training and inference

A method for providing a graphical visualization of a neural network to a user is provided. The method includes generating the graphical visualization of the neural network at least in part by: representing layers of the neural network as respective three-dimensional blocks, wherein at least a first dimension of a given block is proportional to a computational complexity of a layer of the neural network represented by the given block; and representing data flows between the layers of the neural network as respective three-dimensional structures connecting blocks representing the layers of the neural network, wherein a first dimension of a given structure is proportional to each of a first dimension and a second dimension of a data flow represented by the given structure. The method also includes displaying the graphical visualization of the neural network to the user.

Method for visualizing neural network models

A processing unit topology of a neural network including a plurality of processing units is determined. The neural network includes at least one machine in which each machine includes a plurality of nodes, and wherein each node includes at least one of the plurality of processing units. One or more of the processing units are grouped into a first group according to a first affinity. The first group is configured, using a processor and a memory, to use a first aggregation procedure for exchanging model parameters of a model of the neural network between the processing units of the first group. One or more of the processing units are grouped into a second group according to a second affinity. The second group is configured to use a second aggregation procedure for exchanging the model parameters between the processing units of the second group.

Hybrid aggregation for deep learning neural networks

One or more parameter changes for one or more parameters are computed at one or more worker nodes. The one or more parameters on a remote server are updated based on the computed one or more parameter changes. The updating is performed via one or more remote direct memory access atomic operations with the remote server.

Parameter management through RDMA atomic operations

A cache management system performs cache management in a Remote Direct Memory Access (RDMA) key value data store. The cache management system receives a request from at least one client configured to access a data item stored in a data location of a remote server, and determines a popularity of the data item based on a frequency at which the data location is accessed by the at least one client. The system is further configured to determine a lease period of the data item based on the frequency and assigning the lease period to the data location.

Cache management in RDMA distributed key/value stores based on atomic operations

A method includes storing parameter versions utilized by learner instances in each of two or more epochs in a parameter receiving buffer of a parameter server, the learner instances performing distributed execution of workload computations of a machine learning algorithm. The method also includes creating a parameter roster in the parameter server comprising parameter version vectors specifying the parameter versions used by each of the learner instances during each of the two or more epochs. The method further includes generating one or more aggregated parameter sets for storage in an aggregated parameters buffer by aggregating parameter versions from the parameter receiving buffer based on the parameter version vectors in the parameter roster and providing aggregated parameter sets from the aggregated parameters buffer to the learner instances for deterministic replay of the distributed execution of the workload computations of the machine learning algorithm.

Parameter version vectors used for deterministic replay of distributed execution of workload computations

A network interface controller of a machine receives a packet including at least one model parameter of a neural network model from a server. The packet includes a virtual address associated with the network interface controller, and the machine further includes a plurality of graphics processing units coupled to the network interface controller by a bus. The network interface controller translates the virtual address to a memory address associated with each of the plurality of graphics processing units. The network interface controller broadcasts the at least one model parameter to the memory address associated with each of the plurality of graphics processing units.

Local multicast in single-host multi-GPU machine for distributed deep learning systems

At a cache manager of a directed acyclic graph-based data analytic platform, from each of a plurality of monitor components on a plurality of worker nodes, statistics are obtained for a plurality of tasks, including which of the tasks have been processed and which are in a task queue. Each of the tasks has at least one associated distributed dataset. Each worker has a distributed dataset cache. A current stage directed acyclic graph is obtained from a directed acyclic graph scheduler component. For a given one of the tasks which has been processed, and for which it is determined that no other ones of the tasks depend on the at least one distributed dataset associated with the given one of the tasks, the distributed dataset is evicted from a corresponding one of the distributed dataset caches.

Growing community of inventors

Elmsford, NY, United States of America

Yandong Wang