International Business Machines Corporation

Ramot at Tel-aviv University Ltd.

Malte Johannes Rasch

Tayfun Gokmen

Oguzhan Murat Onen

Wilfried Haensch

Seyoung Kim

Effendi Leobandung

Lior Horesh

Chai Wah Wu

Mark Steven Squillante

Tomasz J Nowicki

Vasileios Kalantzis

Xiao Lin Sun

Zhibin Ren

Yulong Li

Geoffrey W Burr

Pritish Narayanan

Haim Avron

HsinYu Tsai

Hyungjun Kim

Mattia Rigotti

A method is presented for artificial neural network training. The method includes storing weight values in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix, defining the weight matrix to have an output dimension that is smaller than the input dimension such that the weight matrix has a rectangular configuration, and converting the weight matrix from a rectangular configuration to a more square-shaped configuration by repeating or concatenating the rectangular configuration of the weight matrix to increase a signal strength of a backward pass signal by copying an input of repeated weight elements during a forward cycle pass, summing output computations from the repeated weight elements, updating each of the repeated weight elements according to a backpropagated error or alternatively updating only one of the repeated weight elements by setting all forward values except one to zero during an update pass.

Weight repetition on RPU crossbar arrays

A computer-implemented method, computer program product, and/or computer system that performs the following operations: (i) receiving a deep neural network (DNN) having a set of inputs nodes, a set of output nodes, and a set of weight parameters; (ii) configuring the DNN for application to a set of analog resistive processing unit (RPU) arrays, the configuring including applying a set of modifiers to respective outputs of the set of output nodes, the set of modifiers corresponding to the set of analog RPU arrays; (iii) training the DNN using a training process, the training yielding an updated set of weight parameters and an updated set of modifiers; and (iv) transferring the updated set of weight parameters and the updated set of modifiers to the set of analog RPU arrays.

Optimizing deep neural network mapping for inference on analog resistive processing unit arrays

Techniques are provided to implement hardware accelerated computing of eigenpairs of a matrix. For example, a system includes a processor, and a resistive processing unit coupled to the processor. The resistive processing unit includes an array of cells which include respective resistive devices, wherein at least a portion of the resistive devices are tunable to encode values of a given matrix which is storable in the array of cells. When the given matrix is stored in the array of cells, the processor is configured to determine an eigenvector of the stored matrix by executing a process which includes performing analog matrix-vector multiplication operations on the stored matrix to converge an initial vector to an estimate of the eigenvector of the stored matrix.

Hardware acceleration for computing eigenpairs of a matrix

Techniques for noise and bound management for DNN training on RPU crossbar arrays using a scaling factor based on a worst-case scenario are provided. In one aspect, a method for noise and bound management includes: obtaining input vector values x for an analog crossbar array of RPU devices, wherein a weight matrix is mapped to the analog crossbar array of RPU devices; and scaling the input vector values x based on a worst-case scenario to provide scaled input vector values x' to use as input to the analog crossbar array of RPU devices, wherein the worst-case scenario includes an assumed maximal weight of the weight matrix multiplied by a sum of absolute values from the input vector values x.

Worst case noise and bound management for RPU crossbar arrays

A device which comprises an array of resistive processing unit (RPU) cells, first control lines extending in a first direction across the array of RPU cells, and second control lines extending in a second direction across the array of RPU cells. Peripheral circuitry comprising readout circuitry is coupled to the first and second control lines. A control system generates control signals to control the peripheral circuitry to perform a first operation and a second operation on the array of RPU cells. The control signals include a first configuration control signal to configure the readout circuitry to have a first hardware configuration when the first operation is performed on the array of RPU cells, and a second configuration control signal to configure the readout circuitry to have a second hardware configuration, which is different from the first hardware configuration, when the second operation is performed on the array of RPU cells.

Dynamic configuration of readout circuitry for different operations in analog resistive crossbar array

A computer implemented method for implementing a convolutional neural network (CNN) using a crosspoint array includes configuring the crosspoint array to implement a convolution layer by storing one or more weights in crosspoint devices of the array. The method further includes making multiple copies of the weights and training the CNN. Training the CNN includes mapping input data of the convolution layer to the crosspoint array in a row-by-row manner. Further the excitation is input in a row-by-row manner into the crosspoint array, thereby creating row-by-row forward output from the crosspoint array. Further, outputs from the crosspoint devices are stored to corresponding integrators. Errors in the outputs as compared to a desired output, from multiple rows are computed and back propagated in a row-by-row manner into the crosspoint array, the computed errors transmitted to a previous convolution layer.

Row-by-row convolutional neural network mapping for analog artificial intelligence network training

A method for multiple copies of a set of multi-kernel set operations in a hardware accelerated neural network includes a word line for receiving a pixel value of an input image. A bit line communicates a modified pixel value. An analog memory cell including a first capacitor stores a first kernel weight of a first kernel in one of a plurality of kernel sets such that the pixel value is operated on by the first kernel weight to produce the modified pixel value. A charge connection connects the first capacitor to at least a second capacitor storing a second kernel weight of a related kernel of a second one of the plurality of kernel sets such that charge is shared between the first capacitor and at least the second capacitor to normalize the first kernel weight and the second kernel weight.

Kernel sets normalization with capacitor charge sharing

Provided are embodiments for a computer-implemented method, a system, and a computer program product for updating analog crossbar arrays. The embodiments include receiving a number used in matrix multiplication to represent using pulse generation for a crossbar array, and receiving a first bit-length to represent the number, wherein the bit-length is a modifiable bit length. The embodiments also include selecting pulse positions in a pulse sequence having the first bit length to represent the number, performing a computation using the selected pulse positions in the pulse sequence, and updating the crossbar array using the computation.

Sparse modifiable bit length deterministic pulse generation for updating analog crossbar arrays

Techniques for auto weight scaling a bounded weight range of RPU devices with the size of the array during ANN training are provided. In one aspect, a method of ANN training includes: initializing weight values win the array to a random value, wherein the array represents a weight matrix W with m rows and n columns; calculating a scaling factor &#x3b2; based on a size of the weight matrix W; providing digital inputs x to the array; dividing the digital inputs x by a noise and bound management factor &#x3b1; to obtain adjusted digital inputs x'; performing a matrix-vector multiplication of the adjusted digital inputs x&#x2032; with the array to obtain digital outputs y&#x2032;; multiplying the digital outputs y&#x2032; by the noise and bound management factor &#x3b1;; and multiplying the digital outputs y&#x2032; by the scaling factor &#x3b2; to provide digital outputs y of the array.

Auto weight scaling for RPUs

Methods and systems of implementing a convolutional neural network are described. In an example, a structure may receive input signals and distribute the input signals to a plurality of unit cells. The structure may include a plurality of multi-kernel modules that may include a respective set of unit cells. A unit cell may correspond to an element of a kernel being implemented in the convolutional neural network and may include a storage component configured to store a weight of a corresponding element of the kernel. A first pass gate of the unit cell may be activated to pass a stored weight of the unit cell to a plurality of operation circuits in the corresponding unit cell, such that the stored weight may be applied to the input signals. The structure may generate a set of outputs based on the application of the stored weights to the input signals.

Multi-kernel configuration for convolutional neural networks

Methods and systems for solving a linear system include setting resistances in an array of settable electrical resistances in accordance with values of an input matrix. A series of input vectors is applied to the array as voltages to generate a series of respective output vectors. Each input vector in the series of vectors is updated based on comparison of the respective output vectors to a target vector. A solution of a linear system is determined that includes the input matrix based on the updated input vectors.

Analog hardware matrix computation

A resistive processing unit cell includes a weight storage device to store a weight value of the resistive processing unit cell, and multiple circuit blocks. Each circuit block includes a weight update circuit coupled to dedicated update control lines, and a weight read circuit coupled to dedicated read control lines. The circuit blocks are configured to operate in parallel to (i) perform separate weight read operations in which each read circuit generates a read current based on a stored weight value, and outputs the read current on the dedicated read control lines of the read circuit, and (ii) perform separate weight update operations in which each update circuit receives respective update control signals on the dedicated update control lines, generates update currents based on the respective update control signals, and applies the update current to the weight storage device to adjust the weight value based on the update current.

Resistive processing unit cell having multiple weight update and read circuits for parallel processing of data using shared weight value

Mechanisms are provided for acceleration of convolutional neural networks on analog arrays. Input ports receive image signals from frames in an input image. Input memory arrays store the image signals received from the input ports into a respective input memory location to create a plurality of image sub-regions in input memory arrays. A distributor associated each of a set of analog array tiles in an analog array to a part of image sub-regions of the input memory arrays, so that one or more of a set of analog memory components is associated with the image signals in a distribution order to create a respective output signal. An assembler stores each of the respective output signals into one of a set of memory outputs in an output order that is determined by the distribution order.

Acceleration of convolutional neural networks on analog arrays

Provided are embodiments for a computer-implemented method, a system, and a computer program product for updating an analog crossbar array. Embodiment include receiving a number used in matrix multiplication to represent using pulse generation for a crossbar array, and receiving a bit-length to represent the number. Embodiments also include selecting pulse positions in a pulse sequence having the bit length to represent the number, performing a computation using the selected pulse positions in the pulse sequence, and updating the crossbar array using the computation.

Pulse generation for updating crossbar arrays

A computer-implemented method for Eigenpair computation is provided. The method includes computing, her a hardware processor, an Eigenvector and respective Eigenvalues of the Eigenvector of a matrix by using a modified Stochastic Optimization process including performing a matrix vector product on a Resistive Processing Unit (RPU) crossbar array operatively coupled to the hardware processor and performing a scalar vector product on a digital device operatively coupled to the hardware processor and representing, for each of an Eigenpair, an initial guess for the Eigenvector and the respective Eigenvalues. The computing step includes storing the matrix in the RPU crossbar array.

Growing community of inventors

Chappaqua, NY, United States of America

Malte Johannes Rasch