Perceive Corporation

Amazon Technologies, Inc.

Percieve Corporation

Brian Thomas

Steven L Teig

Justin Tantiongloc

Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. In some embodiments, the compiler also generates instructions for gating operations. Gating operations, in some embodiments, include gating at multiple levels (e.g., gating of clusters, cores, or memory units). Gating operations conserve power in some embodiments by gating signals so that they do not reach the gated element or so that they are not propagated within the gated element. In some embodiments, a clock signal is gated such that a register that transmits data on a rising (or falling) edge of a clock signal is not triggered.

Compiler for implementing gating functions for neural network configuration

Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. In some embodiments, the graph includes nodes representing options for implementing each layer of the machine-trained network and edges between nodes for different layers representing different implementations that are compatible. In some embodiments, the graph is populated according to rules relating to memory use and the numbers of cores necessary to implement a particular layer of the machine trained network such that nodes for a particular layer, in some embodiments, represent fewer than all the possible groupings of sets of clusters.

Compiler for optimizing number of cores used to implement neural network

Some embodiments provide a method for generating neural network program instructions for a neural network inference circuit to execute a neural network. The neural network inference circuit includes a particular amount of available memory. The method receives a specification of the neural network including multiple layers. The method determines (i) a required amount of weight memory for the neural network and (ii) required amounts of activation memory for each of a set of layers of the neural network. When the required amount of weight memory and the required amount of activation memory for at least one layer is greater than the particular amount of available memory, the method generates the program instructions for the neural network inference circuit to execute a first set of the layers of the neural network multiple times for different blocks of input data and execute a second set of the layers in a single pass.

Generation of machine-trained network instructions

Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. The compiler, as part of generating the graph, in some embodiments, determines whether any set of channels contains no non-zero values (i.e., contains only zero values). For sets of channels that include no non-zero values, some embodiments perform a zero channel removal operation to remove all-zero channels wherever possible. In some embodiments, zero channel removal operations include removing input channels, removing output channels, forward propagation, and backward propagation of channels and constants.

Compiler for performing zero-channel removal

Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. In some embodiments, the graph includes nodes representing options for implementing each layer of the machine-trained network and edges between nodes for different layers representing different implementations that are compatible. The graph is used, in some embodiments, to select an optimum set of cores for implementing the received machine-trained network. The compiler, in some embodiments, optimizes memory storage such that input and output layers of a single layer are not stored in a same memory unit. Such an optimization, in some embodiments, avoids attempting to read and write from a same memory unit within a core in a single clock cycle.

Compiler for optimizing memory allocations within cores

Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). In some embodiments, the compiler determines whether sparsity requirements of channels implemented on individual cores are met on each core. If the sparsity requirement is not met, the compiler, in some embodiments, determines whether the channels of the filter can be rearranged to meet the sparsity requirements on each core and, based on the determination, either rearranges the filter channels or implements a solution to non-sparsity.

Compiler for optimizing filter sparsity for neural network implementation configuration

Some embodiments provide a compiler for optimizing the implementation of a machine-trained network (e.g., a neural network) on an integrated circuit (IC). The compiler of some embodiments receives a specification of a machine-trained network including multiple layers of computation nodes and generates a graph representing options for implementing the machine-trained network in the IC. In some embodiments, the graph includes nodes representing options for implementing each layer of the machine-trained network and edges between nodes for different layers representing different implementations that are compatible. The compiler of some embodiments is also responsible for generating instructions relating to shutting down (and waking up) memory units of cores. In some embodiments, the memory units to shutdown are determined by the compiler based on the data that is stored or will be stored in the particular memory units.

Growing community of inventors

Vancouver, Canada

Brian Thomas