Amazon Technologies, Inc.

Other

Ron Diamant

Randy Renfu Huang

Sundeep Amirineni

Thomas A Volpe

Nafea Bshara

Paul Gilbert Meyer

Richard John Heaton

Kun Xu

Jeffrey T Huynh

Dana Michelle Vantrease

Drazen Borkovic

Erez Izenberg

Ori Weber

Vignesh Vivekraja

Mohammad El-Shabani

Patricio Kaplan

Jindrich Zejda

Alex Levin

Leah Shalev

Ilya Minkin

Georgy Machulsky

Thomas Elmer

Thiam Khean Hah

Kenneth Wayne Patton

Michael Baranchik

Akshay Balasubramanian

Yaniv Shapira

Robert Geva

Raymond Scott Whiteside

Yu Zhou

Elad Valfer

Guy Nakibly

Barak Wasserstrom

Hongbin Zheng

Preston Pengra Briggs

Jonathan Cohen

Haichen Li

Nishith Desai

Joshua Wayne Bowman

Adi Habusha

Andrea Olgiati

Christopher James BeSerra

Dan Trock

Ihab Bishara

Se Jong Oh

Svetlana Kantorovych

Sudipta Sengupta

Nathan Lorenzo Binkert

Mark Banse

Yida Wang

Taylor Goodhart

Taemin Kim

Yizhi Liu

Animesh Jain

Vinod Sharma

Adiel Sarusi

Henry Wang

Noam Efraim Bashari

Noga Smith

Xiaodan Tan

Se Wang Oh

Yair Armoza

Eyal Freund

Omer Shaked

Yair Sandberg

Uday Shilton Udayaselvam

Sagar Sonar

Willis Wang

Matthew John Campagna

James Christopher Sorenson

David James Borland

Zorik Machulsky

Ali Ghassan Saidi

Erez Sabbag

Gil Stoler

Colm Gearóid MacCárthaigh

Saar Gross

Laura J Sharpless

Samuel Jacob

Brian Robert Silver

Amit Pandey

Zhao Ma

Yongseok Koh

Todd Swanson

Lev Makovsky

Michael Butler Fortin

Gary Lee Szilagyi

Benita Bose

Muhannad Ghanem

Sheng Xu

Rashika Kheria

Sunil Kumar Bathula

Tiandong Zhao

Xin Tong

Noa Yehezkel

Ahmad Erew

Systems and methods are provided to improve the memory throughput for storing and reading intermediate data computed by layers of a neural network during a training process. A compression operation can be performed by removing the zeros from the intermediate data and storing locations of the zeros before storing the intermediate data in the memory for a forward pass of the training process. The compressed data can be read from the memory for a backward pass of the training process and de-compressed by inserting zeros based on the stored locations. Additionally, a transpose operation can be performed before compression as a first atomic operation, or after de-compression as a second atomic operation.

Feature-map throughput during training process

An opportunistic approach is described to accelerate certain memory copy operations to be performed by a computing engine by executing instructions. An instruction for a memory copy operation can be identified that has a first data type with a smaller number of bits per data element than supported by the computing engine to perform the copy operation. The instruction can be replaced with another instruction that has a second data type with a higher number of bits per data element based on the alignment of the memory addresses for the copy operation and the total number of data elements to be copied. The second data type may not only accelerate the copy operation but also provide better utilization of the underlying hardware of the computing engine.

Accelerated memory copy operations

Systems and methods for performing hardware approximation of functions are provided. In one example, a system comprises a controller, a plurality of multiplexors, configurable arithmetic circuits, and a mapping table that stores a set of function parameters. According to a mode of operations, the controller may configure the plurality of multiplexors to forward the set of function parameters or a subset of the function parameters to the arithmetic circuits to compute an approximation result. In a case where the subset of the function parameters is forwarded to the arithmetic circuits, the controller may configure the arithmetic circuits to perform post-processing, such as quantization, of the approximation result.

Configurable function approximation based on hardware selection of mapping table content

Techniques for replacing input values being loaded into a computational circuit are described. Small input values such as denormal numbers can be replaced with alternative values such as zeros to reduce switching activity in the computational circuit, and thus reduce power consumption. In applications such as most neural networks, the impact on the prediction results when replacing small numbers with zeros can be negligible. In applications where high precision computations may be desirable, the input values can be loaded into the computation circuit without modification.

Reducing power consumption in integrated circuits

Systems and methods for providing executable instructions to a neural network processor are provided. In one example, a system comprises a database that stores a plurality of executable instructions and a plurality of subgraph identifiers, each subgraph identifier of the plurality of subgraph identifiers being associated with a subset of instructions of the plurality of executable instructions. The system further includes a compiler configured to: identify a computational subgraph from a computational graph of a neural network model; compute a subgraph identifier for the computational subgraph, based on whether the subgraph identifier is included in the plurality of subgraph identifiers, either: obtain, from the database, first instructions associated with the subgraph identifier; or generate second instructions representing the computational subgraph; and provide the first instructions or the second instructions for execution by a neural network processor to perform computation operations for the neural network model.

Neural network processing based on subgraph recognition

Multi-threaded data movement request handling techniques are disclosed. The techniques can include performing, by a direct memory access (DMA) engine, one or more data movement operations associated with a data movement phase of a first data movement task, initiating a data movement phase of a second data movement task during a wait period for the sync phase of the first data movement task, initiating the sync phase of the first data movement task following expiration of the wait period for the sync phase of the first data movement task, and performing a sync operation associated with the sync phase of the first data movement task.

Multi-threaded data movement request handling

Techniques to perform transpose operations in a crossbar circuit may include receiving a set of write transactions to write a data array to a target memory, and determining that the set of write transactions is for a transpose write. Write data for each of the write transactions can be stored diagonally in a transpose memory of the crossbar circuit. Each row of data in the transpose memory can be rotated, and rotated data from each row of the transpose memory can be provided to a corresponding output port of the crossbar circuit to write to the target memory.

Crossbar based transpose data transfers

Systems and methods are provided to perform multiply-accumulate operations of reduced precision numbers in a systolic array. Each row of the systolic array can receive reduced inputs from a respective reducer. The reducer can receive a particular input and generate multiple reduced inputs from the input. The reduced inputs can include reduced input data elements and/or a reduced weights. The systolic array may lack support for inputs with a first bit-length and the reducers may reduce the bit-length of a given input from the first bit-length to a second shorter bit-length and provide multiple reduced inputs with second shorter bit-length to the array. The systolic array may perform multiply-accumulate operations on each unique combination of the multiple reduced input data elements and the reduced weights to generate multiple partial outputs. The systolic array may sum the partial outputs to generate the output.

Systolic array with input reduction to multiple reduced inputs

A computer-implemented method includes receiving a neural network model that includes memory load operations and a plurality of computation operations, selecting a memory load operation having an arithmetic intensity factor (AIF) greater than a threshold value from the memory load operations, grouping computation operations associated with data loaded by the selected memory load operation into two or more clusters of computation operations, and incorporating an instance of the selected memory load operation before each cluster of the two or more clusters of computation operations in the neural network model.

Arithmetic-intensity based load cloning

To improve utilization of a systolic array, each row of the array is provided with two or more general purpose row input buses. Each of the general purpose row input data buses can be operable to transfer either feature map (FMAP) input elements or weight values into the processing elements of the corresponding row of the array. The flexibility of the general purpose row input buses allows weights to be loaded in parallel into the array to speed up weight loading times to reduce the overall latency.

Flexible array data loading

Disclosed herein are techniques for implementing a large fully-connected layer in an artificial neural network. The large fully-connected layer is grouped into multiple fully-connected subnetworks. Each fully-connected subnetwork is configured to classify an object into an unknown class or a class in a subset of target classes. If the object is classified as the unknown class by a fully-connected subnetwork, a next fully-connected subnetwork may be used to further classify the object. In some embodiments, the fully-connected layer is grouped based on a ranking of target classes.

Executing sublayers of a fully-connected layer

Techniques for exchanging compressed gradient data within a distributed system are disclosed. A set of gradients are computed at a first worker node of the distributed system using a neural network model and a set of weights associated with the neural network model. Each of the set of gradients having a value less than a threshold is clipped, resulting in non-clipped data elements and clipped data elements. A mapping indicating which of the set of gradients correspond to non-clipped data elements and which of the set of gradients correspond to clipped data elements is generated. Compressed data is generated based on the non-clipped data elements. The mapping and the compressed data are transmitted from the first worker node to a second worker node of the distributed system.

Gradient compression for distributed training

To reduce power consumption, data bits or a portion of a data register that is not expected to toggle frequently can be grouped together, and be clock-gated independently from the rest of the data register. The grouping of the data bits can be determined based on the data types of the workload being operated on. For a data register configured to store a numeric value that supports multiple data types, the portion of the data register being clock-gated may store a group of data bits that are unused for one or more data types of the multiple data types supported by the data register. The portion of the data register being clock-gated can also be a group of data bits that remain unchanged or have a constant value for numeric values within a certain numeric range that is frequently operated on.

Data-type-aware clock-gating

Provided are systems, methods, and integrated circuits for neural network processing. In various implementations, an integrated circuit for neural network processing can include a plurality of memory banks storing weight values for a neural network. The memory banks can be on the same chip as an array of processing engines. Upon receiving input data, the circuit can be configured to use the set of weight values to perform a task defined for the neural network. Performing the task can include reading weight values from the memory banks, inputting the weight values into the array of processing engines, and computing a result using the array of processing engines, where the result corresponds to an outcome of performing the task.

Multi-memory on-chip computational network

Techniques are described to perform hierarchical collective compute operations on a plurality of processing nodes for a distributed computation of a neural network. Each processing node can include a plurality of processing ranks that are assigned corresponding rank identifiers (IDs). The hierarchical collective compute operation can include direct memory access (DMA) transfers to perform an inter-node collective compute operation of non-contiguous memory regions across the processing nodes based on the rank IDs, and an intra-node collective compute operation of non-contiguous memory regions within each of the processing nodes based on the rank IDs.

Hierarchical collective compute operations using DMA transfers

Growing community of inventors

Albany, CA, United States of America

Ron Diamant