Amazon Technologies, Inc.

Xilinx, Inc.

Hongbin Zheng

Randy Renfu Huang

Richard John Heaton

Drazen Borkovic

Ron Diamant

Animesh Jain

Jeffrey T Huynh

Yida Wang

Yizhi Liu

Haichen Li

Sundeep Amirineni

Stephen Andrew Neuendorffer

Robert Geva

Vinod Sharma

Kecheng Hao

Qingrui Liu

Jindrich Zejda

Vignesh Vivekraja

Preston Pengra Briggs

Amit Kasat

Taemin Kim

Sahil Goyal

Tobias Joseph Kastulus Edler Von Koch

Zhi Chen

Mahesh Attarde

Jiading Gai

Yuwen Jia

Yunxuan Yu

Pushkar Ratnalikar

Thomas Robert Norell

Michael Ray Benfield

Sheng Xu

Jason Xiong

In one example, a method performed by a compiler comprises: receiving a dataflow graph of a neural network, the neural network comprising a neural network operator; receiving information of computation resources and memory resources of a neural network hardware accelerator intended to execute the neural network operator; determining, based on the dataflow graph, iterations of an operation on elements of a tensor included in the neural network operator; determining, based on the information, a mapping between the elements of the tensor to addresses in the portion of the local memory, and a number of the iterations of the operation to be included in a batch, wherein the number of the iterations in the batch are to be executed in parallel by the neural network hardware accelerator; and generating a schedule of execution of the batches of the iterations of the operations.

Global modulo allocation in neural network compilation

Techniques for implementing tensor parallel execution can include identifying a first tensor contraction operation in a compute flow, and slicing the first tensor contraction operation into a first set of multiple tensor contraction portions to have each compute engine of multiple compute engines perform a portion of the first tensor contraction operation. A set of slicing options can then be determined for a second tensor contraction operation that operates on a tensor result of the first tensor contraction operation. A cost for each slicing option is determined, and a slicing option having the lowest cost is selected. The second tensor contraction operation is sliced according to the selected slicing option to have each compute engine perform a portion of the second tensor contraction operation. Collective compute operations can be inserted in the compute flow for the first and second tensor contraction operations.

Compiler managed tensor parallel execution

Techniques for determining a compatible data layout for a computational flow being executed on an integrated circuit device may include obtaining a representation of a set of connected nodes representing loopnests in a compute flow graph. Those of the nodes in the set of connected nodes that are associated with operators having a fixed data layout can be initialized into a set of layout groups. A set of merging operations can then be iteratively performed to form a single layout group from the set of connected nodes. The set of merging operations may include identifying a set of frontier nodes adjacent to the set of layout groups, determining an initialization cost of each of the frontier nodes, selecting a frontier node having a lowest initialization cost, and merging the selected frontier node with one or more adjacent layout groups to form a new layout group having a compatible data layout.

Frontier node-based data layout analysis framework

Generating instructions for programming a processing element array to implement a convolution operation can include determining that the convolution operation under-utilizes the processing element array. The convolution operation involves using the processing element array to perform a series of matrix multiplications between a set of filters and a set of input matrices. Each filter comprises a weight matrix. Each input matrix is assigned to a respective row in the processing element array. Under-utilization can be determined through detecting that less than a threshold number of rows would be used concurrently. In response to determining that the convolution operation under-utilizes the processing element array, instructions can be added for modifying the convolution operation to increase the number of rows used concurrently. The added instructions are executable to cause at least one input matrix to be processed in parallel across more rows compared to processing without modifying the convolution operation.

Efficient utilization of processing element array

Methods and apparatuses for hierarchical partitioning of operators of a neural network for execution on an acceleration engine are provided. Neural networks are built in machine learning frameworks using neural network operators. The neural network operators are compiled into executable code for the acceleration engine. Development of new framework-level operators can exceed the capability to map the newly developed framework-level operators onto the acceleration engine. To enable neural networks to be executed on an acceleration engine, hierarchical partitioning can be used to partition the operators of the neural network. The hierarchical partitioning can identify operators that are supported by a compiler for execution on the acceleration engine, operators to be compiled for execution on a host processor, and operators to be executed on the machine learning framework.

Hierarchical partitioning of operators

A computer-implemented method includes generating, based on a representation of a tensor mapping between an input tensor and an output tensor, a list of mappings from elements of the input tensor to elements of the output tensor, and generating groups of mappings from the list of mappings, where each of the groups of mappings corresponds to a respective set of matrix multiplications, a matrix transpose, or both. The computer-implemented method also includes generating a respective expression for each of the groups of mappings and generating code for summing results of the respective expressions, where each respective expression includes the respective set of matrix multiplications, the matrix transpose, or both.

Converting quasi-affine expressions to matrix operations

A single instruction multiple data (SIMD) processor is used to implement a dropout layer between a first layer and a second layer of a neural network. The SIMD processor can implement the dropout layer by setting one or more elements in an output tensor of the first layer to zero before providing it as an input tensor to the second layer. Setting of the one or more elements to zero is based on a dropout rate, and pseudo-random numbers generated by a random number generator in the SIMD processor.

Dropout layer in a neural network processor

Techniques for reducing a compilation time for compiling a neural network are disclosed. A description of a neural network is received by a compiler. A plurality of operators are identified based on the description of the neural network. A plurality of subgraphs are formed, each including one or more operators. For each subgraph, a performance factor is calculated based on a compute usage and a memory usage associated with the operators included in the subgraph. The performance factor is compared to a threshold. Based on the comparison, either the subgraph is classified as a compute bound subgraph and a set of memory optimizations are suppressed or the subgraph is classified as a memory bound subgraph and a set of compute optimizations are suppressed.

Compilation time reduction for memory and compute bound neural networks

In one example, a method comprises: receiving input codes, wherein the input codes represent a computational dataflow graph; traversing the computational dataflow graph to identify single-entry-single-exit (SESE) subgraphs of the computational dataflow graph, wherein each SESE subgraph has a sequence of nodes comprising a root node and a child node and representing a sequence of element-wise operators, wherein the root node receives a single input tensor, and wherein the child node outputs a single output tensor; determining a merged operator for each SESE subgraph; and generating executable instructions for the computational dataflow graph to be executed by a hardware accelerator having a first execution unit and a second execution unit, wherein the executable instructions comprise first executable instructions for the merged operators targeted at the first execution unit, and second executable instructions for other operators of the computational dataflow graph targeted at the second execution unit.

Reconfigurable neural network processing based on subgraph recognition

Techniques to speed up code compilation may include caching code analysis results such that the analysis of subsequent code having a similar structured can be omitted. For example, a loop-nest construct in the code can be parsed, and an execution statement in the loop-nest construct can be analyzed by a compiler to generate an analysis result indicating a set of execution conditions for the execution statement. A lookup key can be generated from the control statements bounding the execution statement, and the analysis result can be stored with the lookup key in a cache entry of the cache. The execution statement is then modified according to the analysis result for optimization. Instead of having to analyze a subsequent execution statement bounded by the same control statements, the analysis result of the subsequent execution statement can be retrieved from the cache and be used to modify the subsequent execution statement.

Compilation with caching of code analysis result

A computer-implemented method includes receiving a neural network model for implementation using a processing element array, where the neural network model includes a convolution operation on a set of input feature maps and a set of filters. The method also includes determining, based on the neural network model, that the convolution operation utilizes less than a threshold number of rows in the processing element array for applying a set of filter elements to the set of input feature maps, where the set of filter elements includes one filter element in each filter of the set of filters. The method further includes generating, for the convolution operation and based on the neural network model, a first instruction and a second instruction for execution by respective rows in the processing element array, where the first instruction and the second instruction use different filter elements of a filter in the set of filters.

A computer-implemented method includes identifying, from instruction code for executing by a computing system to implement a neural network, a first instruction for allocating a first region of a local memory of an accelerator of the computing system to a tensor, and a first direct memory access (DMA) load instruction for loading the tensor from a location of a system memory of the computing system to a second region of the local memory; adding a first tensor copy instruction in the instruction code to save the tensor in the first region of the local memory to a third region of the local memory that has dimensions different from dimensions of the first region; and replacing the first DMA load instruction with a second tensor copy instruction for saving data in the third region of the local memory to the second region of the local memory.

Growing community of inventors

San Jose, CA, United States of America

Hongbin Zheng