International Business Machines Corporation

Globalfoundries Inc.

Daniel Brand

Minsik Cho

Ruchir Puri

Rajesh Ramkrishna Bordawekar

Ulrich Alfons Finkler

Kailash Gopalakrishnan

Brian R Konigsburg

Jungwook Choi

Naigang Wang

Ankur Agrawal

Vincent Kulandaisamy

John A Darringer

David Shing-ki Kung

Florian Karl Krohm

Silvia Melitta Mueller

Chia-Yu Chen

Vugranam Chakravarthy Sreedhar

Elaine Rivette Palmer

Amitkumar Manoharrao Paradkar

Paul Ashley Karger

Suzanne K McIntosh

Charles L Berman

David Claude Toll

Andrew James Sullivan

Samuel Monk Weber

Matthew N Kaplan

Michael G McIntosh

Marcio Buss

Louse H Trevillyan

Embodiments of the present invention provide a computer-implemented method for adaptive residual gradient compression for training of a deep learning neural network (DNN). The method includes obtaining, by a first learner, a current gradient vector for a neural network layer of the DNN, in which the current gradient vector includes gradient weights of parameters of the neural network layer that are calculated from a mini-batch of training data. A current residue vector is generated that includes residual gradient weights for the mini-batch. A compressed current residue vector is generated based on dividing the residual gradient weights of the current residue vector into a plurality of bins of a uniform size and quantizing a subset of the residual gradient weights of one or more bins of the plurality of bins. The compressed current residue vector is then transmitted to a second learner of the plurality of learners or to a parameter server.

Robust gradient weight compression schemes for deep learning applications

Input image data having a plurality of pixel values represented in a two-dimensional matrix form of columns and rows is received. The input image data is transformed into a plurality of input rows. The pixel values in each input row correspond to the pixel values in a predetermined subset of the columns of the input image data and all of the rows of each column of the subset of columns. A plurality of subsets of pixel values in the plurality of input rows is determined. The number of pixel values in each row of a subset of pixel values equal in number to a number of filter values in a filter. Each input row of each subset of pixel values is convolved with the filter values of the filter to determine a corresponding output value and stored in a memory.

Memory efficient convolution operations in deep learning neural networks

Apparatuses and Methods for sorting a data set. A data storage is divided into a plurality of buckets that is each associated with a respective key value. A plurality of stripes is identified in each bucket. A plurality of data stripe sets is defined that has one stripe within each respective bucket. A first and a second in-place partial bucket radix sort are performed on data items contained within the first and second data stripe sets, respectively, using an initial radix. Incorrectly sorted data items in the first bucket are grouped by a first processor and incorrectly sorted data items in the second bucket are grouped by a second processor into a respective incorrect data item group within each bucket. A radix sort is then performed using the initial radix on the items within the respective incorrect data item group. A first level sorted output is produced.

Parallelized in-place radix sorting

An iterative graph algorithm accelerating method, system, and computer program product, include recording an order of access nodes in a memory layout, reordering the access nodes in the memory layout in accordance with the recorded order, and updating edge information of the reordered access nodes.

System, method and computer program product for accelerating iterative graph algorithms by memory layout optimization

An information processing system, computer readable storage medium, and method for accelerated radix sort processing of data elements in an array in memory. The information processing system stores an array of data elements in a buffer memory in an application specific integrated circuit radix sort accelerator. The array has a head end and a tail end. The system radix sort processing, with a head processor, data elements starting at the head end of the array and progressively advancing radix sort processing data elements toward the tail end of the array. The system radix sort processing, with a tail processor, data elements starting at the tail end of the array and progressively advancing radix sort processing data elements toward the head end of the array, the tail processor radix sort processing data elements in the array contemporaneously with the head processor radix sort processing data elements in the array.

Radix sort acceleration using custom asic

A first quicksort is performed in parallel across pairs of partitions of a dataset assigned to respective ones of available processors, including swapping elements of a first partition of a given one of the pairs that are larger than a pivot with elements of a second partition of the given pair that are smaller than the pivot. A second quicksort is performed in parallel across those partitions having elements left unsorted by the first quicksort, and first misplaced elements from a first side of the dataset corresponding to the first partition are swapped with second misplaced elements from a second side of the dataset corresponding to the second partition to produce a first dataset having elements equal to or lower than the pivot and a second dataset having elements equal to or higher than the pivot.

Parallel quicksort

A compute matrix is configured to include a set of compute units, each compute unit including a multiplier and an accumulator, each of the multiplier and the accumulator formed using at least one floating point unit (FPU). An accumulator array is configured to include a set of external accumulators. The compute matrix is operated to produce a chunk dot-product using a first chunk of a first input vector and a first chunk of a second input vector. The accumulator array is operated to output a dot-product of the first input vector and the second input vector using the chunk dot-product.

Deep learning accelerator architecture with chunking GEMM

A specialized circuit is configured for floating point computations using numbers represented by a very low precision format (VLP format). The VLP format includes less than sixteen bits and is apportion into a sign bit, exponent bits (e), and mantissa bits (p). The configured specialized circuit is operated to store an approximation of a numeric value in the VLP format, where the approximation is represented as a function of a multiple of a fraction, where the fraction is an inverse of a number of discrete values that can be represented using only the mantissa bits.

Very low precision floating point representation for deep learning acceleration

Methods and systems for sorting a dataset include partitioning the dataset into 2partitions, where n is a number of available processors. A first quicksort is performed in parallel across pairs of partitions based on a pivot using a plurality of processors. A second quicksort is performed in parallel on unsorted elements within each partition based on the pivot, where the unsorted elements were left unsorted by the first quicksort. Misplaced elements from a left side of the dataset are swapped with misplaced elements from a right side of the dataset to produce a left dataset that has elements equal to or lower than the pivot and a right dataset that has elements equal to or higher than the pivot.

Radix sort acceleration using custom ASIC

Methods for sorting a data set. A data storage is divided into a plurality of buckets that is each associated with a respective key value. A plurality of stripes is identified in each bucket. At least one data stripe set is defined that has one stripe within each respective bucket. An in-place partial bucket radix sort is performed on data items contained within one data stripe set with a first processor using an initial radix. Incorrectly sorted data items are then grouped in each bucket into a respective incorrect data item group within each bucket. A radix sort is then performed using the initial radix on the items within the respective incorrect data item group. A first level sorted output is produced.

Methods for sorting a data set. Data items each having a first portion and a second portion is stored. The first and second portions are stored separately and each has a separate set of keys. The first portion has a pointer indicating the second portion. At least some of the first set of keys for each data item is stored in a local memory of a first processor. At least one data stripe set is defined with one stripe within each bucket. An in-place partial bucket radix sort is performed on data items within one data stripe set with a first processor using an initial key. Incorrectly sorted data items are grouped into respective incorrect data item groups within each bucket. A radix sort is then performed using the initial radix on the incorrect data item groups. A first level sorted output is produced.

Access is obtained to a truth table having a plurality of rows, each including a plurality of input bits and a plurality of output bits. At least some rows include don't-care inputs. At least some of the rows are clustered into a plurality of multi-row clusters. At least some of the multi-row clusters are assigned to ternary content-addressable memory modules of a prefabricated programmable memory array. Instructions for interconnecting the ternary content-addressable memory modules with a plurality of input pins of the prefabricated programmable memory array and a plurality of output pins of the prefabricated programmable memory array are specified in a data structure, in order to implement the truth table.

Growing community of inventors

Millwood, NY, United States of America

Daniel Brand