International Business Machines Corporation

Yasushi Negishi

Kiyokuni Kawachiya

Tung Duc Le

Taro Sekiyama

Haruki Imai

Jun Doi

Kazuya Tago

Takeshi Ogasawara

Mikio Takeuchi

Yasunao Katayama

Gaku Yamamoto

Atsuya Okazaki

Guojing Cong

Hiroki Murata

Hiroki Nakano

Megumi Ito

I-Hsin Chung

Kun Zhao

Rika Nagahara

Hui-Fang Wen

Kenichi Okuyama

Masaharu Sakamato

Mohan Ahuja

Mikako Hoshiba

Imai Haruki

Atsuya Okaza-Ki

A computer-implemented method, a computer program product, and a computer processing system are provided for selecting from among multiple Graphics Processing Unit (GPU) execution modes for a Neural Network (NN) having a size greater than a threshold size. The multiple GPU execution modes include a normal memory mode, an Out-of-Core (OoC) execution mode, and a Unified Memory (UM) mode. The method includes starting an execution on the NN with the UM mode and measuring the memory usage for each of layers of the NN. The method further includes selecting an execution mode based on the memory usage of all of the layers.

Choosing execution mode of a neural network based on total memory usage

A generated algorithm used by a neural network is captured during execution of an iteration of the neural network. A candidate algorithm is identified based on the generated algorithm. A determination is made that the candidate algorithm utilizes less memory than the generated algorithm. Based on the determination the neural network is updated by replacing the generated algorithm with the candidate algorithm.

Real-time resource usage reduction in artificial neural networks

A method is presented for compressing data of a Rectified Linear Unit (ReLU) function on a graphical processing unit (GPU) employed in a learning process of a deep neural network. The method includes converting an initial data structure including nonzero data and zero data into a compressed data structure including only the nonzero data of the initial data structure as compressed data by generating a nonzero data bitmap region, generating a nonzero data number table region by employing a parallel reduction algorithm, calculating a nonzero data array index per block region of all blocks from the nonzero data number table region by employing a parallel prefix sum scan algorithm, allocating a buffer for the compressed data; and copying the nonzero data from the initial data structure into a nonzero data array region in a compressed data format in parallel.

ReLU compression to reduce GPU memory

Methods and systems for managing data redundancy include registering certified commands, input files, output files, and arguments in an execution history list after execution of said certified commands. An existing output file is provided in response to execution of a first certified command that matches an entry in the execution history list. A file is deleted if the file is reproducible from another file using a second certified command. The deleted file is registered in a reproducible file list. The deleted file is reproduced upon request using the second certified command.

File system for genomic data

A computer-implemented method, computer program product, and computer processing system are provided for accelerating neural network data parallel training in multiple graphics processing units (GPUs) using at least one central processing unit (CPU). The method includes forming a set of chunks. Each of the chunks includes a respective group of neural network layers other than a last layer. The method further includes performing one or more chunk-wise synchronization operations during a backward phase of the neural network data parallel training, by each of the multiple GPUs and the at least one CPU.

Multi-GPU deep learning using CPUs

A method is provided for consistent data processing by first and second distributed processing systems having different data partitioning and routing mechanisms such that the first system is without states and the second system is with states. The method includes dividing data in each system into a same number of partitions based on a same key and a same hash function. The method includes mapping partitions between the systems in a one-to-one mapping. The mapping step includes calculating a partition ID based on the hash function and a total number of partitions, and dynamically mapping a partition in the first system to a partition in the second system, responsive to the partition in the first system being unmapped to the partition in the second system.

Integrating multiple distributed data processing servers with different data partitioning and routing mechanisms, resource sharing policies and lifecycles into a single process

A system and method provides efficient parallel training of a neural network model on multiple graphics processing units. A training module reduces the time and communication overhead of gradient accumulation and parameter updating of the network model in a neural network by overlapping processes in an advantageous way. In a described embodiment, a training module overlaps backpropagation, gradient transfer and accumulation in a Synchronous Stochastic Gradient Decent algorithm on a convolution neural network. The training module collects gradients of multiple layers during backpropagation of training from a plurality of graphics processing units (GPUs), accumulates the gradients on at least one processor and then delivers the gradients of the layers to the plurality of GPUs during the backpropagation of the training. The whole model parameters can then be updated on the GPUs after receipt of the gradient of the last layer.

Efficient parallel training of a network model on multiple graphics processing units

A computer-implemented method is provided for managing GPU memory consumption by computational graph rewriting. The method includes constructing, by a hardware processor, a categorized topological ordering of a computational graph. The categorized topological ordering includes multiple computational nodes arranged in multiple levels. The method further includes estimating, by the hardware processor, the GPU memory consumption responsive to a level including two or more computational nodes from among the multiple computational nodes. The method also includes rewriting, by the hardware processor, the computational graph by linearizing the two or more computational nodes in the level to avoid overlapping of the GPU memory consumption by the two or more computational nodes responsive to the GPU memory consumption exceeding a threshold. The memory additionally includes managing the GPU memory consumption in accordance with the rewritten computational graph.

Graph rewriting for large model support using categorized topological sort

A method is provided for reducing consumption of a memory in a propagation process for a neural network (NN) having fixed structures for computation order and node data dependency. The memory includes memory segments for allocating to nodes. The method collects, in a NN training iteration, information for each node relating to an allocation, size, and lifetime thereof. The method chooses, responsive to the information, a first node having a maximum memory size relative to remaining nodes, and a second node non-overlapped with the first node lifetime. The method chooses another node non-overlapped with the first node lifetime, responsive to a sum of memory sizes of the second node and the other node not exceeding a first node memory size. The method reallocates a memory segment allocated to the first node to the second node and the other node to be reused by the second node and the other node.

Memory reduction for neural networks with fixed structures

A method is provided for consistent data processing by first and second distributed processing systems having different data partitioning and routing mechanisms such that the first system is without states and the second system is with states. The method includes dividing data in each system into a same number of partitions based on a same key and a same hash function. The method includes mapping partitions between the systems in a one-to-one mapping. The mapping step includes (i) checking if a partition in the first system is mapped to a partition in the second system; and (ii) calculating a partition ID based on the hash function and a total number of partitions, and dynamically mapping the partition in the first system to the partition in the second system, responsive to the partition in the first system being unmapped to the partition in the second system.

A computer-implemented method is provided for estimating the performance of a GPU application on a new computing machine having an increased GPU-link performance ratio relative to a current computing machine having a current GPU-link performance ratio. The method includes adding a delay to CPU-GPU communication on the current computing machine to simulate a delayed-communication environment on the current computing machine. The method further includes executing the target GPU application in the delayed-communication environment. The method also includes measuring the performance of the target GPU application in the delayed-communication environment. The method additionally includes estimating the performance of the new computing machine having the increased higher GPU-link performance ratio, based on the measured performance of the target GPU application in the delayed-communication environment.

Estimating performance of GPU application for different GPU-link performance ratio

A target object may be identified by estimating a distribution of a plurality of orientations of a periphery of a target object, and identifying the target object based on the distribution.

Surface-based object identification

A computer-implemented method for data-deduplication of genome data that is in different file formats is described. Representative data from different genome file formats is conformed to a selected file format and compared. Duplicate files are identified and duplicate files are released, with at least one file copy being retained.

Growing community of inventors

Tokyo, Japan

Yasushi Negishi