Amazon Technologies, Inc.

Other

Ron Diamant

Randy Renfu Huang

Sundeep Amirineni

Thomas A Volpe

Nafea Bshara

Richard John Heaton

Paul Gilbert Meyer

Kun Xu

Jeffrey T Huynh

Dana Michelle Vantrease

Drazen Borkovic

Erez Izenberg

Ori Weber

Vignesh Vivekraja

Patricio Kaplan

Jindrich Zejda

Alex Levin

Mohammad El-Shabani

Leah Shalev

Ilya Minkin

Georgy Machulsky

Thomas Elmer

Thiam Khean Hah

Michael Baranchik

Akshay Balasubramanian

Yaniv Shapira

Robert Geva

Raymond Scott Whiteside

Kenneth Wayne Patton

Yu Zhou

Elad Valfer

Guy Nakibly

Barak Wasserstrom

Hongbin Zheng

Preston Pengra Briggs

Jonathan Cohen

Nishith Desai

Haichen Li

Joshua Wayne Bowman

Adi Habusha

Andrea Olgiati

Christopher James BeSerra

Dan Trock

Ihab Bishara

Svetlana Kantorovych

Se Jong Oh

Sudipta Sengupta

Nathan Lorenzo Binkert

Mark Banse

Yida Wang

Taemin Kim

Taylor Goodhart

Yizhi Liu

Animesh Jain

Vinod Sharma

Adiel Sarusi

Noam Efraim Bashari

Yair Armoza

Se Wang Oh

Noga Smith

Xiaodan Tan

Henry Wang

Eyal Freund

Omer Shaked

Willis Wang

Sagar Sonar

Uday Shilton Udayaselvam

Yair Sandberg

Matthew John Campagna

James Christopher Sorenson

David James Borland

Zorik Machulsky

Ali Ghassan Saidi

Erez Sabbag

Gil Stoler

Colm Gearóid MacCárthaigh

Saar Gross

Samuel Jacob

Laura J Sharpless

Brian Robert Silver

Amit Pandey

Zhao Ma

Yongseok Koh

Todd Swanson

Noa Yehezkel

Ahmad Erew

Lev Makovsky

Michael Butler Fortin

Muhannad Ghanem

Benita Bose

Sheng Xu

Gary Lee Szilagyi

Rashika Kheria

Sunil Kumar Bathula

Tiandong Zhao

Techniques for replacing input values being loaded into a computational circuit are described. Small input values such as denormal numbers can be replaced with alternative values such as zeros to reduce switching activity in the computational circuit, and thus reduce power consumption. In applications such as most neural networks, the impact on the prediction results when replacing small numbers with zeros can be negligible. In applications where high precision computations may be desirable, the input values can be loaded into the computation circuit without modification.

Reducing power consumption in integrated circuits

Systems and methods for providing executable instructions to a neural network processor are provided. In one example, a system comprises a database that stores a plurality of executable instructions and a plurality of subgraph identifiers, each subgraph identifier of the plurality of subgraph identifiers being associated with a subset of instructions of the plurality of executable instructions. The system further includes a compiler configured to: identify a computational subgraph from a computational graph of a neural network model; compute a subgraph identifier for the computational subgraph, based on whether the subgraph identifier is included in the plurality of subgraph identifiers, either: obtain, from the database, first instructions associated with the subgraph identifier; or generate second instructions representing the computational subgraph; and provide the first instructions or the second instructions for execution by a neural network processor to perform computation operations for the neural network model.

Neural network processing based on subgraph recognition

Multi-threaded data movement request handling techniques are disclosed. The techniques can include performing, by a direct memory access (DMA) engine, one or more data movement operations associated with a data movement phase of a first data movement task, initiating a data movement phase of a second data movement task during a wait period for the sync phase of the first data movement task, initiating the sync phase of the first data movement task following expiration of the wait period for the sync phase of the first data movement task, and performing a sync operation associated with the sync phase of the first data movement task.

Multi-threaded data movement request handling

Techniques to perform transpose operations in a crossbar circuit may include receiving a set of write transactions to write a data array to a target memory, and determining that the set of write transactions is for a transpose write. Write data for each of the write transactions can be stored diagonally in a transpose memory of the crossbar circuit. Each row of data in the transpose memory can be rotated, and rotated data from each row of the transpose memory can be provided to a corresponding output port of the crossbar circuit to write to the target memory.

Crossbar based transpose data transfers

Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and/or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Systolic array with input reduction to multiple reduced inputs

A computer-implemented method includes receiving a neural network model that includes memory load operations and a plurality of computation operations, selecting a memory load operation having an arithmetic intensity factor (AIF) greater than a threshold value from the memory load operations, grouping computation operations associated with data loaded by the selected memory load operation into two or more clusters of computation operations, and incorporating an instance of the selected memory load operation before each cluster of the two or more clusters of computation operations in the neural network model.

Arithmetic-intensity based load cloning

To improve utilization of a systolic array, each row of the array is provided with two or more general purpose row input buses. Each of the general purpose row input data buses can be operable to transfer either feature map (FMAP) input elements or weight values into the processing elements of the corresponding row of the array. The flexibility of the general purpose row input buses allows weights to be loaded in parallel into the array to speed up weight loading times to reduce the overall latency.

Flexible array data loading

Disclosed herein are techniques for implementing a large fully-connected layer in an artificial neural network. The large fully-connected layer is grouped into multiple fully-connected subnetworks. Each fully-connected subnetwork is configured to classify an object into an unknown class or a class in a subset of target classes. If the object is classified as the unknown class by a fully-connected subnetwork, a next fully-connected subnetwork may be used to further classify the object. In some embodiments, the fully-connected layer is grouped based on a ranking of target classes.

Executing sublayers of a fully-connected layer

Techniques for exchanging compressed gradient data within a distributed system are disclosed. A set of gradients are computed at a first worker node of the distributed system using a neural network model and a set of weights associated with the neural network model. Each of the set of gradients having a value less than a threshold is clipped, resulting in non-clipped data elements and clipped data elements. A mapping indicating which of the set of gradients correspond to non-clipped data elements and which of the set of gradients correspond to clipped data elements is generated. Compressed data is generated based on the non-clipped data elements. The mapping and the compressed data are transmitted from the first worker node to a second worker node of the distributed system.

Gradient compression for distributed training

To reduce power consumption, data bits or a portion of a data register that is not expected to toggle frequently can be grouped together, and be clock-gated independently from the rest of the data register. The grouping of the data bits can be determined based on the data types of the workload being operated on. For a data register configured to store a numeric value that supports multiple data types, the portion of the data register being clock-gated may store a group of data bits that are unused for one or more data types of the multiple data types supported by the data register. The portion of the data register being clock-gated can also be a group of data bits that remain unchanged or have a constant value for numeric values within a certain numeric range that is frequently operated on.

Data-type-aware clock-gating

Provided are systems, methods, and integrated circuits for neural network processing. In various implementations, an integrated circuit for neural network processing can include a plurality of memory banks storing weight values for a neural network. The memory banks can be on the same chip as an array of processing engines. Upon receiving input data, the circuit can be configured to use the set of weight values to perform a task defined for the neural network. Performing the task can include reading weight values from the memory banks, inputting the weight values into the array of processing engines, and computing a result using the array of processing engines, where the result corresponds to an outcome of performing the task.

Multi-memory on-chip computational network

Techniques are described to perform hierarchical collective compute operations on a plurality of processing nodes for a distributed computation of a neural network. Each processing node can include a plurality of processing ranks that are assigned corresponding rank identifiers (IDs). The hierarchical collective compute operation can include direct memory access (DMA) transfers to perform an inter-node collective compute operation of non-contiguous memory regions across the processing nodes based on the rank IDs, and an intra-node collective compute operation of non-contiguous memory regions within each of the processing nodes based on the rank IDs.

Hierarchical collective compute operations using DMA transfers

A technique to program a compute channel having multiple computational circuit blocks coupled in series in a pipeline can include receiving a machine instruction for the compute channel. The machine instruction is decoded to obtain an opcode, and the opcode can be used as an index to access an opcode entry in an opcode table. The opcode entry contains a pointer to a microoperation, and the pointer can be used to access a microoperation represented by a control entry in a control table and a datapath configuration entry in a datapath table. The microoperation can then be issued to the compute channel by configuring the compute channel with the control entry and the datapath configuration entry.

Configuration of a deep vector engine using an opcode table, control table, and datapath table

A compute channel can have multiple computational circuit blocks coupled in series to form a pipeline. The compute channel can perform a computation on an input tensor to generate an output tensor based on an instruction. When the computational does not require all of the computational circuit blocks, the throughput of the compute channel can be increased by splitting the data elements of the input tensor into multiple input data streams. The multiple input data streams are provided to respective subsets of one or more computational circuit blocks in the pipeline using bypass circuitry of the computational circuit blocks, and the computation can be performed on multiple input data streams in the respective subsets of one or more computational circuit blocks to generate multiple output data streams corresponding to the output tensor.

Throughput increase for compute engine

To reduce the storage size of weight tensors and speed up loading of weight tensors from system memory, a compression technique can be employed to remove zero values from a weight tensor before storing the weight tensor in system memory. A sparsity threshold can be enforced to achieve a compression ratio target by forcing small weight values to zero during training. When the weight tensor is loaded from system memory, a direct memory access (DMA) engine with an in-line decompression unit can decompress the weight tensor on-the-fly. By performing the decompression in the DMA engine, expansion of the weight values back to the original weight tensor size can be carried out in parallel while other neural network computations are being performed by the processing unit.

Sparse machine learning acceleration

Growing community of inventors

Albany, CA, United States of America

Ron Diamant