Perceive Corporation

Alexandru F Drimbarean

Steven L Teig

Eric A Sather

Some embodiments of the invention provide a method for configuring a network with multiple nodes. Each node generates an output value based on received input values and a set of weights that are previously trained to each have an initial value. For each weight, the method calculates a factor that represents a loss of accuracy to the network due to changing the weight from its initial value to a different value in a set of allowed values for the weight. Based on the factors, the method identifies a subset of the weights that have factors with values below a threshold. The method changes the values of each weight from its initial value to one of the values in its set of allowed values. The values of the identified subset are all changed to zero. The method trains the weights beginning with the changed values for each weight.

Initialization of values for training a neural network with quantized weights

Some embodiments of the invention provide a method for configuring a machine-trained (MT) network that includes multiple nodes. Each node of a set of the nodes generates an output value based on received input values and a set of configurable weights. The method propagates a set of inputs through the MT network to generate a set of outputs, with each input having a corresponding expected output. The method calculates a value of a loss function comprising (i) a first term that measures a difference between each generated output and its corresponding expected output and (ii) a second term that constrains the weights to discrete sets of allowed values and accounts for an increase in the first term due to constraining the weights to the discrete sets of values. The method uses the calculated value of the loss function to train the weights of the MT network.

Training a neural network with quantized weights

Some embodiments provide a method that receives a network with trained floating-point weight values. The network includes layers of nodes, each of which computes an output value based on input values and trained weight values. To replace a first layer of the trained network in a modified network with quantized weight values, the method defines multiple replica layers. Each replica layer includes nodes that correspond to nodes of the first layer, has a different set of allowed quantized weight values, and receives the same input values from a previous layer of the modified network such that groups of corresponding nodes from the replica layers operate correspondingly to the first layer. The method trains the quantized weight values of the modified network using a loss function with terms that account for effect on the loss function due to the quantization and for interactions between corresponding weight values of the replica layers.

Growing community of inventors

Galway, Ireland

Alexandru F Drimbarean