Samsung Electronics Co., Ltd.

Weiran Deng

Georgios Georgiadis

Zhengping Ji

A technique to prune weights of a neural network using an analytic threshold function h(w) provides a neural network having weights that have been optimally pruned. The neural network includes a plurality of layers in which each layer includes a set of weights w associated with the layer that enhance a speed performance of the neural network, an accuracy of the neural network, or a combination thereof. Each set of weights is based on a cost function C that has been minimized by back-propagating an output of the neural network in response to input training data. The cost function C is also minimized based on a derivative of the cost function C with respect to a first parameter of the analytic threshold function h(w) and on a derivative of the cost function C with respect to a second parameter of the analytic threshold function h(w).

Self-pruning neural networks for weight parameter reduction

A method of manufacturing an apparatus is provided. The apparatus is formed on a wafer or a package. The apparatus includes a polynomial generator, a plurality of matrix generators connected to an output of the polynomial generator, and a convolution generator connected to an output of the plurality of matrix generators. The apparatus is tested using one or more electrical to optical converters, one or more optical splitters, and one or more optical to electrical converters.

Apparatus and method for generating efficient convolution

A system and a method to normalize a deep neural network (DNN) in which a mean of activations of the DNN is set to be equal to about 0 for a training batch size of 8 or less, and a variance of the activations of the DNN is set to be equal to about a predetermined value for the training batch size. A minimization module minimizes a sum of a network loss of the DNN plus a sum of a product of a first Lagrange multiplier times the mean of the activations squared plus a sum of a product of a second Lagrange multiplier times a quantity of the variance of the activations minus one squared.

Normalization method for training deep neural networks

A system and a method generate a neural network that includes at least one layer having weights and output feature maps that have been jointly pruned and quantized. The weights of the layer are pruned using an analytic threshold function. Each weight remaining after pruning is quantized based on a weighted average of a quantization and dequantization of the weight for all quantization levels to form quantized weights for the layer. Output feature maps of the layer are generated based on the quantized weights of the layer. Each output feature map of the layer is quantized based on a weighted average of a quantization and dequantization of the output feature map for all quantization levels. Parameters of the analytic threshold function, the weighted average of all quantization levels of the weights and the weighted average of each output feature map of the layer are updated using a cost function.

Jointly pruning and quantizing deep neural networks

A method includes: providing a deep neural networks (DNN) model comprising a plurality of layers, each layer of the plurality of layers includes a plurality of nodes; sampling a change of a weight for each of a plurality of weights based on a distribution function, each weight of the plurality of weights corresponds to each node of the plurality of nodes; updating the weight with the change of the weight multiplied by a sign of the weight; and training the DNN model by iterating the steps of sampling the change and updating the weight. The plurality of weights has a high rate of sparsity after the training.

Method for optimizing neural networks

A system and method that provides balanced pruning of weights of a deep neural network (DNN) in which weights of the DNN are partitioned into a plurality of groups, a count of a number of non-zero weights is determined in each group, a variance of the count of weights in each group is determined, a loss function of the DNN is minimized using Lagrange multipliers with a constraint that the variance of the count of weights in each group is equal to 0, and the weights and the Lagrange multipliers are retrained by back-propagation.

Method to balance sparsity for efficient inference of deep neural networks

A system and method for pruning. A neural network includes a plurality of long short-term memory cells, each of which includes an input having a weight matrix W, an input gate having a weight matrix W, a forget gate having a weight matrix W, and an output gate having a weight matrix W. In some embodiments, after initial training, one or more of the weight matrices W, W, and Ware pruned, and the weight matrix Wis left unchanged. The neural network is then retrained, the pruned weights being constrained to remain zero during retraining.

Accelerating long short-term memory networks via selective pruning

A method of manufacturing an apparatus and a method of constructing an integrated circuit are provided. The method of manufacturing an apparatus includes forming the apparatus on a wafer or a package with at least one other apparatus, wherein the apparatus comprises a polynomial generator, a first matrix generator, a second matrix generator, a third matrix generator, and a convolution generator; and testing the apparatus, wherein testing the apparatus comprises testing the apparatus using one or more electrical to optical converters, one or more optical splitters that split an optical signal into two or more optical signals, and one or more optical to electrical converters.

An apparatus and a method is provided. The apparatus includes a polynomial generator, including an input and an output; a first matrix generator, including an input connected to the output of the polynomial generator, and an output; a second matrix generator, including an input connected to the output of the first matrix generator, and an output; a third matrix generator, including a first input connected to the output of the first matrix generator, a second input connected to the output of the second matrix generator, and an output; and a convolution generator, including an input connected to the output of the third matrix generator, and an output.

Growing community of inventors

Woodland Hills, CA, United States of America

Weiran Deng