International Business Machines Corporation

Globalfoundries Inc.

University of Texas System

Minsik Cho

Ruchir Puri

Ulrich Alfons Finkler

Daniel Brand

Haoxing Ren

Brian R Konigsburg

Rajesh Ramkrishna Bordawekar

David Shing-ki Kung

David Shing-Ki Kung

Hua Xiang

Matthew Mantell Ziegler

Jeonghee Shin

Indira Nair

Vincent Kulandaisamy

Li Feng Zhang

Wei Zhang

Vladimir P Zolotov

Xiaoping Tang

Bumsoo Kang

Inseok Hwang

Chin Ngai Sze

Jinho Lee

Seetharami R Seelam

Haifeng Qian

Vinod Muthusamy

Alvan Wing Ng

Louise Helen Trevillyan

Jagannathan Narasimhan

Zhigang Pan

Andrew James Sullivan

Victor N Kravets

Frank Liu

Mayoore Selvarasa Jaiswal

Dorothy Kucar

Smita Krishnaswamy

Chungkuk Yoo

Soham Shah

A batched Cholesky decomposition method, system, and non-transitory computer readable medium for a Graphics Processing Unit (GPU), include mirroring matrices to form paired matrices solving the paired matrices simultaneously.

Mirroring matrices for batched cholesky decomposition on a graphic processing unit

An overall gradient vector is computed at a server from a set of ISA vectors corresponding to a set of worker machines. An ISA vector of a worker machine including ISA instructions corresponding to a set of gradients, each gradient corresponding to a weight of a node of a neural network being distributedly trained in the worker machine. A set of register values is optimized for use in an approximation computation with an opcode to produce an x-th approximate gradient of an x-th gradient. A server ISA vector is constructed in which a server ISA instruction in an x-th position corresponds to the x-th gradient in the overall gradient vector. A processor at the worker machine is caused to update a set of weights of the neural network, using the set of optimized register values and the server ISA vector, thereby completing one iteration of training.

ISA-based compression in distributed training of neural networks

Techniques that facilitate model support in deep learning are provided. In one example, a system includes a graphics processing unit and a central processing unit memory. The graphics processing unit processes data to train a deep neural network. The central processing unit memory stores a portion of the data to train the deep neural network. The graphics processing unit provides, during a forward pass process of the deep neural network that traverses through a set of layers for the deep neural network from a first layer of the set of layers to a last layer of the set of layers that provides a set of outputs for the deep neural network, input data for a layer from the set of layers for the deep neural network to the central processing unit memory.

Large model support in deep learning

A computer-implemented method comprises linking a private AI model to a public AI model to thereby form a combined AI model comprising the private AI model and the public AI model; and training the combined AI model with private samples while keeping the public AI model fixed so that only the private AI model is trained with the private samples.

Separating public and private knowledge in AI

System, method, and recording medium for mirroring matrices for batched cholesky decomposition on a graphic processing unit

Various embodiments are provided for decentralized distributed deep learning by one or more processors in a computing system. Asynchronous distributed training of one or more machine learning models may be performed by generating a list of neighbor nodes for each node in a plurality of nodes and creating a first thread for continuous communication according to a weight management operation and a second thread for continuous computation of a gradient for each node. One or more variables are shared between the first thread and the second thread.

Decentralized distributed deep learning

In deep learning, and in particular, for data compression for allreduce in deep learning, a gradient may be compressed for synchronization in a data parallel deep neural network training for allreduce by sharing a consensus vector between each node in a plurality of nodes to ensure identical indexing in each of the plurality of nodes prior to performing sparse encoding.

Cloud computing data compression for allreduce in deep learning

A computer-implemented method generates a hamming code based target label for each class of a dataset in which hamming distance between the target labels in the dataset is maximized and trains a convolutional neural network with the hamming codes based target label to thereby produce a trained AI model. The confusability between classes of the dataset is determined using a confusion matrix. The hamming distances of classes of the dataset that are determined to be more confusable are set to higher values than the hamming distances of classes of the dataset that are determined to be less confusable.

Hamming distance based robust output encoding for improved generalization

A computer-implemented method comprises: inputting into an autoencoder sets of input samples, each of the sets of input samples comprising: a reference input sample of a reference dataset and one or more target input samples of one or more target datasets, the autoencoder being trained using the reference dataset. The autoencoder generates a respective set of outputs for each set of the input samples to thereby form one or more respective sets of outputs, each of the one or more sets of outputs comprising the reference output and the one or more target outputs for a respective set of input samples; and determining the similarity of each of the one or more target datasets to the reference dataset by comparing each of the one or more target outputs to respective target input samples of each of the sets of input samples.

Method to measure similarity of datasets for given AI task

Input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows is received. The input image data is transformed into a plurality of input rows. The pixel values in each input row correspond to the pixel values in a predetermined subset of the columns of the input image data and all of the rows of each column of the subset of columns. A plurality of subsets of pixel values in the plurality of input rows is determined. The number of pixel values in each row of a subset of pixel values equal in number to a number of filter values in a filter. Each input row of each subset of pixel values is convolved with the filter values of the filter to determine a corresponding output value and stored in a memory.

Memory efficient convolution operations in deep learning neural networks

Matrix factorization based gradient compression may be applied to an allreduce operation to improve efficiency including the elimination of unnecessary meta data while maintaining accuracy in training of deep learning (DL) of Artificial Intelligence. This compression may include generating a predetermined matrix and a degree of data compression k as a dimension of the predetermined matrix for a plurality of computing nodes. Each computing node may receive a corresponding matrix of matrices to be allreduced, and each corresponding matrix may be decomposed into a plurality of non-fixed matrices and the predetermined matrix. The plurality of non-fixed matrices may be summed to provide an optimized matrix, which may be multiplied by the predetermined matrix to provide a result matrix. The optimized matrix may be designated as a predetermined matrix. These operations may be repeated until all of the matrices received by the computing nodes have been allreduced.

Matrix-factorization based gradient compression

Embodiments for pipelining multi-directional reduction by one or more processors in a computing system. One or more reduce scatter operations and one or more all-gather operations may be assigned to each of a plurality of independent networks. The one or more reduce scatter operations and the one or more all-gather operations may be sequentially executed in each of the plurality of independent networks according to a serialized execution order and a defined time period.

Pipelining multi-directional reduction

Using a processor and a memory at a worker machine, a gradient vector is computed corresponding to a set of weights associated with a set of nodes of a neural network instance being trained in the worker machine. In an ISA vector corresponding to the gradient vector, an ISA instruction is constructed corresponding to a gradient in a set of gradients in the gradient vector, wherein a data transmission of the ISA instruction is smaller as compared to a data transmission of the gradient. The ISA vector is transmitted from the worker machine to a parameter server, the ISA vector being responsive to one iteration of a training of the neural network instance, the ISA vector being transmitted instead of the gradient vector to reduce an amount of data transmitted from the worker machine to the parameter server for the one iteration of the training.

Variable ISA vector-based compaction in distributed training of neural networks

A batched Cholesky decomposition method, system, and non-transitory computer readable medium for a Graphics Processing Unit (GPU), include mirroring a second problem matrix of a second problem to a first problem matrix of a first problem as paired matrices and shifting the second problem matrix by N+1 and combining the first problem matrix and the mirrored second problem matrix into one matrix of (N+1)&#xd7;N, where the first problem shared memory comprises regular intervals, where the second problem shared memory is continuous, and where the GPU performs batched dense Cholesky decomposition with the one matrix from the combining to accelerate the Cholesky decomposition.

System, method, and recording medium for mirroring matrices for batched Cholesky decomposition on a graphic processing unit

Growing community of inventors

Austin, TX, United States of America

Minsik Cho