Neuralmagic Inc.

Justin Kopinsky

Nir N Shavit

Alexander Matveev

Mark Kurtz

Dan Alistarh

Rati Gelashvili

Aleksandar Zlateski

A system and method of accelerating execution of a NN model, by at least one processor may include: receiving a first matrix A, representing elements of a kernel K of the NN model and a second matrix B, representing elements of an input I to kernel K; producing from matrix A, a group-sparse matrix A', comprising G tensors of elements. The number of elements in each tensor is defined by, or equal to a number of entries in each index of an input tensor register used for a specific Single Instruction Multiple Data (SIMD) tensor operation, and all elements of A&#x2032; outside said G tensors are null. The system and method may further include executing kernel K on input I, by performing at least one computation of the SIMD tensor operation, having as operands elements of a tensor of the G tensors and corresponding elements of the B matrix.

System and method of accelerating execution of a neural network

A system and method of executing a convolution layer of a neural network may include: (a) selecting an output spatial position (OSP) of an output matrix data element of the convolution layer; (b) selecting, based on the selected OSP, a non-zero input element of an input matrix data element; (c) producing, based on the selected OSP, a vector of kernel elements from a kernel matrix data element; (d) performing a vectoral multiplication operation of the selected non-zero input element and the vector of kernel elements, and accumulating a product of the vectoral multiplication in a vector register of a processor; (e) repeating (c) and (d) with subsequent non-zero input elements and corresponding vectors of kernel elements to obtain an outcome of the convolution of the selected OSP; and (f) repeating (a) through (e) with subsequent selection of OSPs, to obtain an outcome of the convolution layer.

System and method for executing convolution in a neural network

A system and method may generate code to be used when executing neural networks (NNs), for example convolutional neural networks (CNNs) which may include one or more convolutional layers. For at least one convolutional layer, for each non-zero element in a kernel tensor or matrix associated with the convolutional layer, instructions may be generated or issued. For example, for each non-zero element, a vector broadcast instruction may be generated, and a fused multiply-add (FMA) instruction may be generated, having as parameters a register representing a portion of the output for the convolutional layer, a register storing input data for the convolutional layer, and a register or reference to memory storing the non-zero element. The software or code produced may be executed during convolutional operations, for example as part of a larger application such as a NN inference application.

Growing community of inventors

Cambridge, MA, United States of America

Justin Kopinsky