Meta Platforms, Inc.

Maxim Naumov

Jong Soo Park

Abdulkadir Utku Diril

Andrew John Tulloch

Benjamin Ray

Jedrzej Jablonski

The disclosed computer-implemented method may include (1) identifying an artificial neural network comprising a set of nodes interconnected via a set of connections, and (2) training the artificial neural network by, for each connection in the set of connections, determining a quantized weight value associated with the connection. Determining the quantized weight value associated with the connection may include (1) associating a loss function with the connection, the loss function including a periodic regularization function that describes a relationship between an input value and a weight value of the connection, (2) determining a minimum of the associated loss function with respect to the weight value in accordance with the periodic regularization function, and (3) generating the quantized weight value associated with the connection based on the determined minimum of the loss function. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for quantizing neural networks via periodic regularization functions

A computer-implemented method for translating with limited attention a sequence of input tokens into a sequence of output tokens may include (1) encoding by an encoder recurrent neural network (RNN) N input tokens of a sequence of (N+D) input tokens into an attention matrix limited to N input tokens, (2) iteratively updating a previous version of the attention matrix with the context vector generated from each excess input token yielding a final attention matrix at the last excess input token (D), and (3) decoding an output token using a decoder RNN with inputs from at least the weighted context vector and the just previously-determined output token of the decoder. The weighted context vector may be derived from an attention RNN processing the elements of the final attention matrix for each of the decoder output tokens. Various other methods, systems, and computer-readable media are also disclosed.

Growing community of inventors

Redwood City, CA, United States of America

Maxim Naumov