International Business Machines Corporation

Vijayalakshmi Srinivasan

Swagath Venkataramani

Kailash Gopalakrishnan

Alper Buyuktosunoglu

Brian Robert Prasky

Jungwook Choi

Philip George Emma

Jude A Rivers

Sunil K Shukla

Pradip Bose

Harold Wade Cain, Iii

Hillery C Hunter

Chia-Yu Chen

David Michael Daly

Thomas R Puzak

Bruce Martin Fleischer

Xiaochen Guo

Allan Mark Hartstein

Erik Richter Altman

Jintao Zhang

Jinwook Oh

Jason Zebchuk

Brian William Curran

Shubham Jain

Victor Han

Peter Anthony Sandon

Zhigang Hu

Arnold S Tran

Moinuddin Khalil Ahmed Qureshi

Sanchari Sen

Galen A Rasche

Christian Jacobi

Anthony Saporito

Leland Chang

Alan G Gara

Robert K Montoye

Cedric Lichtenau

James J Bonanno

Martin Recktenwald

Pak-kin Mak

Hans Mikael Jacobson

Martin Ohmacht

Alexandre E Eichenberger

Erik L Hedberg

Craig Raymond Walters

Timothy Carl Bronson

Garrett M Drapala

Victor V Zyuban

Wing Kin Luk

John-David Wellman

Ankur Agrawal

Moinuddin K Qureshi

Willm Hinrichs

Rebecca Marie Gott

Martin A Lutz

Michael K Gschwind

Eric M Schwarz

Luis Alfonso Lastras-Montano

Silvia Melitta Mueller

Arvind Kumar

Eren Kursun

Philip Heidelberger

John Edward Barth, Jr

Michele M Franceschini

Ravi Nair

William Robert Reohr

Stanley Everett Schuster

Chen-Yong Cher

Thomas Pflueger

Xiaowei Shen

Prabhakar Nandavar Kudva

Wei Wang

John Timothy Robinson

Daniel Arthur Prener

Peter William Cook

Yulong Li

Klaus J Getzlaff

Marc Raymond Faucher

Lakshminarayanan Renganarayana

Razvan Peter Figuli

Peter Howland Oden

Ioana Monica Baldini Soares

Holger Horbach

John E Barth

Mark Jay Charney

Khubaib Khubaib

Islam Atta

David M Brooks

Kailash Gopalakrishan

Lee M Saltzman

Ioana Monica Burcea

Sarada Krithivasan

Embodiments are provided for efficient realization of memory-bound operations in a computing system by a processor. Data may be read from and written to a memory at a granular level using a stickification operation. One or more regions of activation and weight tensor data on the memory may be annotated by coupling the stickification operation with padding.

Stickification using anywhere padding to accelerate data manipulation

Processing input data for transmittal to a data consumer such as an artificial intelligence engine is performed by arranging the input data into a uniform structure made up of sticks of data combined to form pages of sticks. A stick is any well-sized set of input data elements whereby the size of the stick is fixed. A masking pattern is established for sticks of data having certain ranges of invalid data for consumption of partial sticks while maintaining validity of the input data being transferred. The mask pattern is derived based on set-active-mask-and-value (SAMV) instructions. The derived mask pattern is carried forward for subsequent load instructions to the data consumer.

Padding input data for artificial intelligence accelerators

A combined function specified by an instruction is performed. The combined function includes a plurality of operations performed as part of one invocation of the combined function. The performing the combined function includes performing a matrix multiplication of a first tensor and a second tensor to obtain one or more intermediate results. The second tensor includes an adjusted weight tensor created using a multiplier. Values of a bias tensor are added to the one or more intermediate results to obtain one or more results for the combined function. The one or more results are at least a part of an output tensor.

Single function to perform combined matrix multiplication and bias add operations

A method for improving performance of a predefined Deep Neural Network (DNN) convolution processing on a computing device includes inputting parameters, as input data into a processor on a computer that formalizes a design space exploration of a convolution mapping, on a predefined computer architecture that will execute the predefined convolution processing. The parameters are predefined as guided by a specification for the predefined convolution processing to be implemented by the convolution mapping and by a microarchitectural specification for the processor that will execute the predefined convolution processing. The processor calculates performance metrics for executing the predefined convolution processing on the computing device, as functions of the predefined parameters, as proxy estimates of performance of different possible design choices to implement the predefined convolution processing.

Method to map convolutional layers of deep neural network on a plurality of processing elements with SIMD execution units, private memories, and connected as a 2D systolic processor array

A memory system, a method of assembling the memory system, and a computer system. The memory system includes a global memory device coupled to a plurality of processing elements. The global memory device is positioned external to a chip on which the plurality of processing devices reside. The memory system also includes at least one main scratchpad coupled to the at least one processing element of the plurality of processing devices and the global memory device. The memory system further includes a plurality of auxiliary scratchpads coupled to the plurality of processing elements and the global memory device. The one or more auxiliary scratchpads are configured to store static tensors. At least a portion of the plurality of auxiliary scratchpads are configured as a unitary multichannel device.

Multichannel memory to augment local memory

A compensated deep neural network (compensated-DNN) is provided. A first vector having a set of components and a second vector having a set of corresponding components are received. A component of the first vector includes a first quantized value and a first compensation instruction, and a corresponding component of the second vector includes a second quantized value and a second compensation instruction. The first quantized value is multiplied with the second quantized value to compute a raw product value. The raw product value is compensated for a quantization error according to the first and second compensation instructions to produce a compensated product value. The compensated product value is added into an accumulated value for the dot product. The accumulated value is converted into an output vector of the dot product. The output vector includes an output quantized value and an output compensation instruction.

Low precision deep neural network enabled by compensation instructions

Indices of non-zero weights may be stored in an index register file included within each of a plurality of processor elements in a systolic array. Non-zero weights may be stored in a register file associated with the index register file. Input values (e.g., dense input values) corresponding to a single block in a data structure may be sent to the plurality of processor elements. Those of the input values corresponding to the indices of non-zero weights in the index register file may be selected for performing multiply-accumulate ('MAC') operation based on sending the plurality of input values to one or more of the plurality of processor elements. The indices of the plurality of non-zero weight are stored in an index data stick. The values of the plurality of non-zero weights are stored in a value data stick.

Exploiting fine-grained structured weight sparsity in systolic arrays

Embodiments for providing enhanced multicast data transfer for ring topology based artificial intelligence systems are disclosed. Multicast data is sent to a plurality of disjointed cores in a multicast group according to a first multicast mode, a second multicast mode, or a third multicast mode, where the first multicast mode sends a first half the multicast data on first multicast ring and a second half on a second multicast ring, the second multicast mode sends the multicast data on either the first multicast ring or the second multicast ring, and the third multicast mode replicates the multicast data and sends the multicast data to both the first multicast ring and the second multicast ring.

Programmable multicast protocol for ring-topology based artificial intelligence systems

A systolic array can be configured to skip distributed operands that have zero-values, resulting in improved resource efficiency. A skip module is introduced to receive operands from memory, identify whether they have a zero value or not, and, if they are nonzero, generate an operand vector including an index before sending the operand vector to a processing element.

Sparse systolic array design

Various embodiments are provided reusing an operand in an instruction set architecture (ISA) by one or more processors in a computing system. An instruction may specify that an operand register for a selected operand retain operand data used by a previous instruction. The operand data in the operand register may be reused by the instruction.

Reusing an operand received from a first-in-first-out (FIFO) buffer according to an operand specifier value specified in a predefined field of an instruction

The embodiments herein describe hybrid parallelism techniques where a mix of data and model parallelism techniques are used to split the workload of a layer across an array of processors. When configuring the array, the bandwidth of the processors in one direction may be greater than the bandwidth in the other direction. Each layer is characterized according to whether they are more feature heavy or weight heavy. Depending on this characterization, the workload of an NN layer can be assigned to the array using a hybrid parallelism technique rather than using solely the data parallelism technique or solely the model parallelism technique. For example, if an NN layer is more weight heavy than feature heavy, data parallelism is used in the direction with the greater bandwidth (to minimize the negative impact of weight reduction) while model parallelism is used in the direction with the smaller bandwidth.

Hybrid data-model parallelism for efficient deep learning

A convolutional neural network includes a front layer, a back layer, and a plurality of other layers that are connected between the front layer and the back layer. One of the other layers is a transition layer. A first precision is assigned to activations of neurons from the front layer back to the transition layer and a second precision is assigned to activations of the neurons from the transition layer back to the back layer. A third precision is assigned to weights of inputs to neurons from the front layer back to the transition layer and a fourth precision is assigned to weights of inputs to the neurons from the transition layer back to the back layer. In some embodiments the layers forward of the transition layer have a different convolutional kernel than the layers rearward of the transition layer.

System-aware selective quantization for performance optimized distributed deep learning

A minibatch in a neural network execution may be dynamically resized based on on-chip memory. For example, a size of the minibatch is configured such that the minibatch fits within on-chip memory. The size of the minibatch may be resized for a sequence of layers in the neural network execution. A next layer's execution can commence responsive to the resized minibatch being completed in a previous layer without having to wait for all of the minibatch to be completed in the previous layer.

Dynamically resizing minibatch in neural network execution

A reduced precision based programmable and single instruction multiple data (SIMD) dataflow architecture includes reduced precision execution units with a majority of the execution units operating at reduced precision and a minority of the execution units are capable of operating at higher precision. The execution units operate in parallel within a programmable execution element to share instruction fetch, decode, and issue pipelines and operate on the same instruction in lock-step to minimize instruction-related overhead.

Reduced precision based programmable and SIMD dataflow architecture

A method is provided for forming a Deep Neural Network (DNN). The method includes quantizing deep learning data structures of the DNN into at least two modes using at least two scale factors, respectively. Each of the at least two modes corresponds to a respective one of the at least two scale factors. The method further includes identifying which of the at least two scale factors to use for a given one of the data structures based on a data distribution of the given one of the data structures. The quantizing step includes identifying when a tail of the given one of the data structures starts by (i) building a histogram of values in the given one of the data structures using successive bins; (ii) identifying a ratio of density between the successive bins; and (iii) checking whether the ratio of density is greater than a ratio of density threshold.

Bi-scaled deep neural networks

Growing community of inventors

New York, NY, United States of America

Vijayalakshmi Srinivasan