Amazon Technologies, Inc.

Intel Corporation

Thiam Khean Hah

Randy Renfu Huang

Vignesh Vivekraja

Ron Diamant

Paul Gilbert Meyer

Richard John Heaton

Vamsi Nalluri

Carl Ebeling

Herman Henry Schmit

Scott J Weber

Jason Gee Hock Ong

Yeong Tat Liew

Jiading Gai

Yongseok Koh

In one example, a method comprises: performing backward propagation computations for a second layer of a neural network to generate second weight gradients; splitting the second weight gradients into a plurality of subsets each associated with a second exchange operation over a computer network, a number of the second weight gradients included in the each subset being based on at least one of first characteristics of the computer network or second characteristics of the neural network; performing backward propagation computations for a first layer of the neural network to generate first weight gradients in parallel with at least one of the second exchange operations; performing a first exchange operation to exchange the first weight gradients after the at least one of the second exchange operations completes; and after the first exchange operation completes, perform the remaining second exchange operations to exchange the remaining subsets of the second weight gradients.

Neural network training in a distributed system

Techniques for implementing bidirectional networks, such as bidirectional recurrent neural networks (RNNs), on data-flow centric processors are disclosed. An input to a bidirectional network is provided. The bidirectional network includes a forward network and a reverse network. The forward network performs a set of operations in a forward direction and the reverse network performs the set of operations in a reverse direction. A forward output is generated using the forward network based on the input. A first transformation is performed on the input to generate a transformed input. A reverse output is generated using the reverse network based on the transformed input. A second transformation is performed on the reverse output to generate a transformed output. The forward output and the transformed output are combined to produce an overall output.

Bidirectional network on a data-flow centric processor

Systems and methods are disclosed to perform gradient exchange among processing nodes configured as a hyper-rectangle network of N-dimensions. Each processing node can operate as a collective parameter server node capable to perform collective compute operations. For each dimension in a sequence of dimensions, all processing nodes on a same edge can perform a scatter-reduce operation using respective collective parameter serving engines. The amount of data reduced in each dimension is an inverse of a number of processing nodes in that dimension. After the scatter-reduce operation is performed for all the dimensions, all processing nodes on the same edge can perform an all-gather operation for each dimension in a reverse sequence of dimensions.

Hyper-rectangle network for gradient exchange

Accelerated convolution of neural networks can be performed by executing N computing engines (CEs) of a neural network processor in parallel. An input dataset can be divided spatially into N chunks such that a respective last portion of each chunk overlaps with a respective first portion of a subsequent chunk. Portions of each chunk can be processed by a respective CE to generate a respective portion of an output dataset. The overlapping intermediate states computed by each CE from processing the overlapping portion can be stored locally for sharing with a subsequent CE using an on-chip bus.

Accelerated convolution of neural networks

A systolic array can implement an architecture tailored to perform matrix multiplications on sparse matrices. Each processing element in the systolic array may include a register configured to store a value, and a multiplexor configured to select an input element from multiple input data buses based on metadata associated with the value. Each processing element may also include a multiplier configured to multiply the selected input element with the value to generate a multiplication result, and an adder configured to add the multiplication result to a partial sum input to generate a partial sum output.

Fine-grained sparsity computations in systolic array

To take advantage of the architecture of a systolic array tailored to perform sparse matrix multiplications, a weight matrix can be converted into a set of constrained fine-grained sparse weight matrices. The conversion process may include receiving a request to perform a matrix multiplication operation with a weight matrix, and determining that the weight matrix satisfies a sparsity condition to convert the weight matrix into a set of constrained fine-grained sparse weight matrices. The weight matrix can then be converted into a set of constrained fine-grained sparse weight matrices. Computer instructions can then be generated for an integrated circuit device to perform the requested matrix multiplication operation as a set of sparse matrix multiplication operations using the set of constrained fine-grained sparse weight matrices.

Emulating fine-grained sparsity in a systolic array

A single neural network model can be used by each computing engine (CE) in a neural network processor to perform convolution operations in parallel for one or more stacks of convolutional layers. An input feature map can be divided into N chunks to be processed by N CEs, respectively. Each CE can process a last portion of a respective chunk to generate respective shared states to be used by a subsequent CE. A first CE uses pre-computed states to generate a first portion of an output feature map, while other CEs use shared states computed by a preceding CE to generate respective portions of the output feature map.

Acceleration of neural networks with stacks of convolutional layers

Methods and systems for performing a training operation of a neural network are provided. In one example, a method comprises: performing backward propagation computations for a second layer of a neural network to generate second weight gradients; splitting the second weight gradients into portions; causing a hardware interface to exchange a first portion of the second weight gradients with the second computer system; performing backward propagation computations for a first layer of the neural network to generate first weight gradients when the exchange of the first portion of the second weight gradients is underway, the first layer being a lower layer than the second layer in the neural network; causing the hardware interface to transmit the first weight gradients to the second computer system; and causing the hardware interface to transmit the remaining portions of the second weight gradients to the second computer system.

A systolic array can implement an architecture tailored to perform matrix multiplications on constrained fine-grained sparse weight matrices. Each processing element in the systolic array may include a weight register configured to store a weight value, and a multiplexor configured to select a feature map (FMAP) input element from multiple FMAP input data buses based on metadata associated with the weight value. Each processing element may also include a multiplier configured to multiply the selected feature map input element with the weight value to generate a multiplication result, and an adder configured to add the multiplication result to a partial sum input to generate a partial sum output.

An integrated circuit that includes common factor mass multiplier (CFMM) circuitry is provided that multiplies a common factor operand by a large number of multiplier operands. The CFMM circuitry may be implemented as a instance specific version (where at least some portion of the hardware has to be redesigned if the multipliers change) or a non-instance specific version (where the CFMM circuitry can work with arbitrary multipliers without having to redesign the hardware). Either version can be formed on a programmable integrated circuit or an application-specific integrated circuit. The CFMM circuitry may include a multiplier circuit that effectively multiplies the common factor by predetermined fixed constants to generate partial products and may further include shifting and add/subtract circuits for processing and combining the partial products to generate corresponding final output products. CFMM circuitry configured in this way can be used to support convolution neural networks or any operation that requires a straight common factor multiply.

Common factor mass multiplication circuitry

Systems and methods are included for efficiently implementing learned parameter systems (LPSs) on a programmable integrated circuit (PIC) via a computing engine. The computing engine receives an input set of learned parameters corresponding to use instances of an LPS. The computing engine reduces at least some redundancies and/or unnecessary operations using instance specific parameter values of the LPS, to generate a less redundant set of learned parameters and a corresponding less redundant LPS. The computing engine generates a netlist based on these, which may share computing resources of the PIC across multiple computations in accordance with the less redundant set of learned parameters and the corresponding less redundant LPS. The computing engine then programs the PIC with the netlist. That is, the netlist replaces use instances of at least some of the original learned parameters and its corresponding LPS and is executed instead of the original.

Systems and methods for implementing learned parameter systems on a programmable integrated circuit

An integrated circuit that includes common factor mass multiplier (CFMM) circuitry is provided that multiplies a common factor operand by a large number of multiplier operands. The CFMM circuitry may be implemented as an instance specific version or a non-instance specific version. The instance specific version might also be fully enumerated so that the hardware doesn't have to be redesigned assuming all possible unique multiplier values are implemented. Either version can be formed on a programmable integrated circuit or an application-specific integrated circuit. CFMM circuitry configured in this way can be used to support convolution neural networks or any operation that requires a straight common factor multiply. Any adder component with the CFMM circuitry may be implemented using bit-serial adders. The bit-serial adders may be further connected in a tree in CNN applications to sum together many input streams.

Growing community of inventors

Milpitas, CA, United States of America

Thiam Khean Hah