Xilinx, Inc.

Lsi Corporation

Lsi Logic Corporation

Other

Intel Corporation

Ephrem C Wu

Xiaoqian Zhang

Ting Zhou

David Berman

Sheng Liu

Robin J Tang

Tezaswi Raja

Elliott Delaye

Hongbin Ji

Xiao Teng

Nikola Radovanovic

William James Dally

John Wood Poulton

James Karp

John H Edmondson

Rafael C Camarota

Vassili Kireev

Michael O Jenkins

Ye Liu

Raghunandan Chaware

Wei-Je Robert Huang

John M Thendean

Ehsan Ghasemi

Adam Elkins

Steven J Pollock

Vojislav Vukovic

Thomas H Strader

Donald A Priore

Rajeev Patwari

Jorn Tuyls

Bahareh Banijamali

Gyanesh Saharia

Sanket Pandit

Freeman V Zhong

Hyuck Jin Kwon

Brian Vanderwarn

Inkeun Cho

Robert Hong

Yashodhara Parulkar

Adnan Pratama

Hardware acceleration of machine learning (ML) designs includes translating an ML primitive into an intermediate representation. The intermediate representation is subdivided to specify a functional compute block. The functional compute block is sized according to a compute node primitive adapted for implementing the ML primitive on target hardware. An overlay is generated for the ML primitive, at least in part, by mapping the functional compute block to the compute node primitive. The overlay is synthesizable to implement the ML primitive on the target hardware. The overlay can be scheduled for operation within the target hardware as part of an ML design including the ML primitive.

Hardware acceleration of machine learning designs

Examples herein describe a hardware accelerator for affine transformations (matrix multiplications followed by additions) using an outer products process. In general, the hardware accelerator reduces memory bandwidth by computing matrix multiplications as a sum of outer products. Moreover, the sum of outer products benefits parallel hardware that accelerates matrix multiplication, and is compatible with both scalar and block affine transformations, and more generally, both scalar and block matrix multiplications.

Hardware acceleration of affine transformations

A non-blocking crossbar switch architecture circumvents the problem present in prior art crossbar switches where input signals may oversubscribe the available inter-die bandwidth. The new non-blocking crossbar switch architecture is split across a plurality of semiconductor dice, including a plurality of interleaved crossbar switch segments. Only one crossbar switch segment is implemented on each semiconductor die. A plurality of input ports and output ports are coupled to the crossbar switch. The crossbar switch is non-blocking, i.e., any one output port not currently receiving data may receive data from any one input port.

Multi-die non-blocking crossbar switch

A digital processing engine is configured to receive input data from a memory. The input data comprises first input channels. The digital processing engine is further configured to convolve, with a convolution model, the input data. The convolution model comprises a first filter layer configured to generate first intermediate data having first output channels. A number of the first output channels is less than a number of the first input channels. The convolution model further comprises a second filter layer comprising shared spatial filters and is configured to generate second intermediate data by convolving each of the first output channels with a respective one of the shared spatial filters. Each of the shared spatial filters comprises first weights. The digital processing engine is further configured to generate output data from the second intermediate data and store the output data in the memory.

Shared depthwise convolution

Examples herein describe techniques for performing parallel processing using a plurality of processing elements (PEs) and a controller for data that has data dependencies. For example, a calculation may require an entire row or column to be summed, or to determine its mean. The PEs can be assigned different chunks of a data set (e.g., a tensor set, a column, or a row) for processing. The PEs can use one or more tokens to inform the controller when they are done with partial processing of their data chunks. The controller can then gather the partial results and determine an intermediate value for the data set. The controller can then distribute this intermediate value to the PEs which then re-process their respective data chunks using the intermediate value to generate final results.

Scalable acceleration of reentrant compute operations

Embodiments herein describe circuitry with improved efficiency when executing layers in a nested neural network. As mentioned above, a nested neural network has at least one split operation where a tensor generated by a first layer is transmitted to, and processed by several branches in the neural network. Each of these branches can have several layers that have data dependencies which result in a multiply-add array sitting idly. In one embodiment, the circuitry can include a dedicated pre-pooler for performing a pre-pooling operation. Thus, the pre-pooling operation can be performing in parallel with other operations (e.g., the convolution performed by another layer). Once the multiply-add array is idle, the pre-pooling operation has already completed (or at least, has already started) which means the time the multiply-add array must wait before it can perform the next operation is reduced or eliminated.

Neural-network pooling

Disclosed circuits and methods involve a first register configured to store of a first convolutional neural network (CNN) instruction during processing of the first CNN instruction and a second register configured to store a second CNN instruction during processing of the second CNN instruction. Each of a plurality of address generation circuits is configured to generate one or more addresses in response to an input CNN instruction. Control circuitry is configured to select one of the first CNN instruction or the second CNN instruction as input to the address generation circuits.

Neural network controller

A circuit arrangement includes an array of MAC circuits, wherein each MAC circuit includes a cache configured for storage of a plurality of kernels. The MAC circuits are configured to receive a first set of data elements of an IFM at a first rate. The MAC circuits are configured to perform first MAC operations on the first set of the data elements and a first one of the kernels associated with a first OFM depth index during a first MAC cycle, wherein a rate of MAC cycles is faster than the first rate. The MAC circuits are configured to perform second MAC operations on the first set of the data elements and a second one of the kernels associated with a second OFM depth index during a second MAC cycle that consecutively follows the first MAC cycle.

Performing consecutive mac operations on a set of data using different kernels in a MAC circuit

The embodiments herein store tabulated values representing a linear or non-linear function in separate memory banks to reduce the size of memory used to store the tabulated values while being able to provide upper and lower values for performing linear interpolation in parallel (e.g., the same cycle). To do so, a linear interpolation system includes a first memory bank that stores the even indexed tabulated values while a second memory bank stores the odd indexed tabulated values. During each clock cycle, the first and second memory banks can output upper and lower values for linear interpolation (although which memory bank outputs the upper value and which outputs the lower value can vary). Using the upper and lower values, the linear interpolation system performs linear interpolation to approximate the value of a non-linear function that is between the upper and lower values.

Linear interpolator of tabulated functions

An integrated circuit (IC) includes a plurality of dies. The IC includes a plurality of memory channel interfaces configured to communicate with a memory, wherein the plurality of memory channel interfaces are disposed within a first die of the plurality of dies. The IC may include a compute array distributed across the plurality of dies and a plurality of remote buffers distributed across the plurality of dies. The plurality of remote buffers are coupled to the plurality of memory channels and to the compute array. The IC further includes a controller configured to determine that each of the plurality of remote buffers has data stored therein and, in response, broadcast a read enable signal to each of the plurality of remote buffers initiating data transfers from the plurality of remote buffers to the compute array across the plurality of dies.

Data transfers between a memory and a distributed compute array

A digital signal processor (DSP) slice is disclosed. The DSP slice includes an input stage to receive a plurality of input signals, a pre-adder coupled to the input stage and configured to perform one or more operations on one or more of the plurality of input signals, and a multiplier coupled to the input stage and the pre-adder and configured to perform one or more multiplication operations on one or more of the plurality of input signals or the output of the pre-adder. The DSP slice further includes an arithmetic logic unit (ALU) coupled to the input stage, the pre-adder, and the multiplier. The ALU is configured to perform one or more mathematical or logical operations on one or more of the plurality of input signals, the output of the pre-adder, or the output of the multiplier. The DSP slice also includes an output stage coupled to the ALU, the output stage configured to generate one or more output signals based at least in part on one or more of the outputs of the ALU, or at least one of the plurality of input signals.

Digital signal processing block

Disclosed circuitry includes RAM circuits, a memory controller, and an array of processing circuits. Each RAM circuit includes a read port and a write port. The memory controller accesses tensor data arranged in banks of tensor buffers in the RAM circuits. The memory controller is coupled to each read port by shared read control signal lines and to each write port by shared write control signal lines. The memory controller generates read control and write control signals for accessing different ones of the tensor buffers at different times. The array of processing circuits is coupled to one of the RAM circuits. The array includes multiple rows and multiple of columns of processing circuits for performing tensor operations on the tensor data. The processing circuits in each row in each array of processing circuits are coupled to input the same tensor data.

Memory arrangement for tensor data

An example semiconductor device includes a first integrated circuit (IC) die including a first column of cascade-coupled resource blocks; a second IC die including a second column of cascade-coupled resource blocks, where an active side of the second IC die is mounted to an active side of the first IC die; and a plurality of electrical connections between the active side of the first IC and the active side of the second IC, the plurality of electrical connections including at least one electrical connection between the first column of cascade-coupled resource blocks and the second column of cascade-coupled resource blocks.

Stacked columnar integrated circuits

A memory device can be used with a shared routing resource that provides access to the memory device. The memory device can include a random access memory (RAM) circuit that includes a plurality of ports configured to provide access to the RAM circuit by the shared routing resource. A memory partition register circuit can be configured to store a plurality of addresses specifying respective context partitions within the RAM circuit. A plurality of pointer register circuits that can each be associated with a corresponding port of the plurality of ports and can be configured to store a respective set of pointers that specify a location in the RAM circuit relative to a respective context partition. Addressing logic that can be configured to provide access to the RAM circuit using the respective set of pointers for each port.

Context driven memory circuits

A system includes global memory circuitry configured to store input tensors and output tensors. Row data paths are each connected to an output port of the memory circuitry. Column data paths are connected to an input port of the memory circuitry. Processing elements are arranged in rows and columns along the row data paths and column data paths, respectively. The processing elements include local memory circuitry configured to store multiple masks and processing circuitry. The processing circuitry is configured to receive portions of the input tensors from one of the row data paths; receive masks from the local memory circuitry; perform multiple tensor operations on a same received portion of an input tensors by applying a different retrieved mask for each tensor operation; and generate, using results of the multiple tensor operations, an output for a corresponding column data path.

Tensor operations and acceleration

Growing community of inventors

San Mateo, CA, United States of America

Ephrem C Wu