Amazon Technologies, Inc.

Lg Electronics Inc.

Tariq Afzal

Arvind Mandhani

Thomas Zou

Seungyoon Peter Song

Arkadi Avrukin

Milan Shah

Michael Frank

Hoshik Kim

Yongjae Hong

Jungsook Lee

Shiva Navab

Described herein is a neural network accelerator (NNA) with reconfigurable memory resources for forming a set of local memory buffers comprising at least one activation buffer, at least one weight buffer, and at least one output buffer. The NNA supports a plurality of predefined memory configurations that are optimized for maximizing throughput and reducing overall power consumption in different types of neural networks. The memory configurations differ with respect to at least one of a total amount of activation, weight, or output buffer memory, or a total number of activation, weight, or output buffers. Depending on which type of neural network is being executed and the memory behavior of the specific neural network, a memory configuration can be selected accordingly.

Neural network accelerator with reconfigurable memory

Described herein is a neural network accelerator (NNA) with a decompression unit that can be configured to perform multiple types of decompression. The decompression may include a separate subunit for each decompression type. The subunits can be coupled to form a pipeline in which partially decompressed results generated by one subunit are input for further decompression by another subunit. Depending on which types of compression were applied to incoming data, any number of the subunits may be used to produce a decompressed output. In some embodiments, the decompression unit is configured to decompress data that has been compressed using a zero value compression scheme, a shared value compression scheme, or both. The NNA can also include a compression unit implemented in a manner similar to that of the decompression unit.

Decompression and compression of neural network data using different compression schemes

The present disclosure relates to a system and method for optimizing switching of a DRAM bus using LLC. An embodiment of the disclosure includes sending a first type request from a first type queue to the second memory via the memory bus if a direction setting of the memory bus is in a first direction corresponding to the first type request, decrementing a current direction credit count by a first type transaction decrement value, if the decremented current direction credit count is greater than zero, sending another first type request to the second memory via the memory bus and decrementing the current direction credit count again by the first type transaction decrement value, and if the decremented current direction credit count is zero, switching the direction setting of the memory bus to a second direction and resetting the current direction credit count to a second type initial value.

System and method for optimizing DRAM bus switching using LLC

Described herein is a neural network accelerator with a set of neural processing units and an instruction set for execution on the neural processing units. The instruction set is a compact instruction set including various compute and data move instructions for implementing a neural network. Among the compute instructions are an instruction for performing a fused operation comprising sequential computations, one of which involves matrix multiplication, and an instruction for performing an elementwise vector operation. The instructions in the instruction set are highly configurable and can handle data elements of variable size. The instructions also implement a synchronization mechanism that allows asynchronous execution of data move and compute operations across different components of the neural network accelerator as well as between multiple instances of the neural network accelerator.

Neural network accelerator with compact instruct set

A control circuit for controlling memory prefetch requests to system level cache (SLC). The control circuit includes a circuit identifying memory access requests received at the system level cache (SLC), where each of the memory access requests includes an address (ANEXT) of memory to be accessed. Another circuit associates a tracker with each of the memory access streams. A further circuit performs tracking for the memory access streams by: when the status is tracking and the address (ANEXT) points to an interval between the current address (ACURR) and the last prefetched address (ALAST), issuing a prefetch request to the SLC; and when the status is tracking, and distance (ADIST) between the current address (ACURR) and the last prefetched address (ALAST) is greater than a specified maximum prefetch for the associated tracker, waiting for further requests to control a prefetch process.

Growing community of inventors

San Jose, CA, United States of America

Tariq Afzal