Hewlett Packard Enterprise Development LP

Cong Xu

Paolo Faraboschi

Qiong Cai

Naveen Muralimanohar

Harumi Kuno

Andrew Christopher Walton

Sai Rahul Chalamalasetti

Ping Chi

Erik Ordentlich

Sergey Serebryakov

Tsung Ching Huang

Itir Akgun

Kaisheng Ma

Sicheng Li

Systems and methods are provided for data shuffling for distributed machine learning training, including each training node in the network receiving a shard of training data, wherein the training data set is divided into shards having data items. Each data item is assigned to a working set such that each of the working set includes data items from multiple shards. The training nodes perform training using the data items of a first working set that are in each node's shard. Upon completion of the training using the data items of the first working set, the training nodes performing training using the data items of a second working set that are in their shards; and while the training nodes are performing training on their respective subsets of shards of the second working set, the nodes randomly shuffling data items in the first working set to create a shuffled first working set.

Systems and methods for intelligent data shuffling for high-performance distributed machine learning training

A technique includes modifying a neural network model to sparsify the model. The model includes a plurality of kernel element weights, which are parameterized according to a plurality of dimensions. Modifying the model includes, in a given iteration of the plurality of iterations, training the model based on a structure regularization in which kernel element weights that share a dimension in common are removed as a group to create corresponding zero kernel elements in the model; and compressing the model to exclude zero kernel element weights from the model to prepare the model to be trained in another iteration.

Sparsifying neural network models

While scheduled checkpoints are being taken of a cluster of active compute nodes distributively executing an application in parallel, a likelihood of failure of the active compute nodes is periodically and independently predicted. Responsive to the likelihood of failure of a given active compute node exceeding a threshold, the given active compute node is proactively migrated to a spare compute node of the cluster at a next scheduled checkpoint. Another spare compute node of the cluster can perform prediction and migration. Prediction can be based on both hardware events and software events regarding the active compute nodes.

Proactive cluster compute node migration at next checkpoint of cluster upon predicted node failure

Examples disclosed herein relate to using a memory side accelerator to calculate updated deep learning parameters. A globally addressable memory includes deep learning parameters. The deep learning parameters are partitioned, where each partition is associated with a memory side accelerator. A memory side accelerator is to receive calculated gradient updates associated with its partition and calculate an update to the deep learning parameters associated with the partition.

Memory side acceleration for deep learning parameter updates

Proactive cluster compute node migration at next checkpoint of cluster cluster upon predicted node failure

In some examples, with respect to adaptive multi-level checkpointing, a transfer parameter associated with transfer of checkpoint data from a node-local storage to a parallel file system may be ascertained for the checkpoint data stored in the node-local storage. The transfer parameter may be compared to a specified transfer parameter threshold. A determination may be made, based on the comparison of the transfer parameter to the specified transfer parameter threshold, as to whether to transfer the checkpoint data from the node-local storage to the parallel file system.

Adaptive multi-level checkpointing

Techniques for reallocating a memory pending queue based on stalls are provided. In one aspect, it may be determined at a memory stop of a memory fabric that at least one class of memory access is stalled. It may also be determined at the memory stop of the memory fabric that there is at least one class of memory access that is not stalled. At least a portion of a memory pending queue may be reallocated from the class of memory access that is not stalled to the class of memory access that is stalled.

Reallocate memory pending queue based on stall

Examples described herein include receiving an operation pipeline for a computing system and building a graph that comprises a model for a number of potential memory side accelerator thread assignments to carry out the operation pipeline. The computing system may comprise at least two memories and a number of memory side accelerators. Each model may comprise a number of steps and at least one step out of the number of steps in each model may comprise a function performed at one memory side accelerator out of the number of memory side accelerators. Examples described herein also include determining a cost of at least one model.

Memory side accelerator thread assignments

In an example, in a method for encoding data within a crossbar memory array containing cells, bits of input data may be received. The received bits of data may be mapped to the cells in a row of the memory array, in which the cells are to be assigned to one of a low resistance state and a high resistance state. A subset of the mapped bits in the row may be grouped into a word pattern. The word pattern may be arranged such that more low resistance states are mapped to cells that are located closer to a voltage source of the row of the memory array than to cells that are located farther away from the voltage source.

Growing community of inventors

Palo Alto, CA, United States of America

Cong Xu