Google Inc.

Aurko Roy

Rohan Anil

Noam M Shazeer

Yonghui Wu

Jakob David Uszkoreit

Lukasz Mieczyslaw Kaiser

Andrew M Dai

Samuel Bengio

Niki J Parmar

Ashish Teku Vaswani

Melvin Jose Johnson Premkumar

Ian Goodfellow

Orhan Firat

Slav Petrov

Yanping Huang

Yuanzhong Xu

David Richard So

Nan Du

Colin Abraham Raffel

Maxim Krikun

Gaurav Mishra

Xavier Eduardo Garcia

Vladimir Feinberg

Jacob Buckman

Dmitry Lepikhin

Siamak Shakeri

Yong Cheng

Jonathan Hudson Clark

Erica Ann Moreira

Yujing Zhang

Zachary Alexander Nado

Kefan Xiao

Jin Young Sohn

Eric Jun Jie Ni

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes a neural network configured to perform the machine learning task, the neural network comprising an N-grammer layer and an output neural network, the N-grammer layer configured to: at each of one or more heads: receive a sequence of input embeddings; generate a discrete latent representation of the sequence of input embeddings by using a learned product quantization codebook; generate a plurality of n-gram indices from the discrete latent representation; and generate a latent n-gram representation of the sequence of input embeddings; and generate a sequence of output embeddings, and the output neural network configured to: receive the sequence of output embeddings; and process the sequence of output embeddings to generate the network output.

Attention neural networks with N-grammer layers

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes receiving the input sequence; processing the input sequence using a latent prediction model configured to autoregressively predict a sequence of discrete latent variables that is shorter than the output sequence and that encodes the output sequence, wherein each discrete latent variable in the sequence is selected from a discrete set of latent variables; and processing the input sequence and the predicted sequence of discrete latent variables using a parallel decoder model configured to generate the outputs in the output sequence in parallel from the input sequence and the predicted sequence of discrete latent variables.

Fast decoding in sequence models using discrete latent variables

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform any one or more of a variety of machine learning tasks. For example, the neural network can be configured as a generative neural network, e.g., an autoregressive generative neural network.

Training of large neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for increasing the security of neural network by discretizing neural network inputs. One of the methods includes receiving a network input for a neural network; processing the network input using a discretization layer, wherein the discretization layer is configured to generate a discretized network input comprising a respective discretized vector for each of the numeric values in the network input; and processing the discretized network input using the plurality of additional neural network layers to generate a network output for the network input.

Increasing security of neural networks by discretizing neural network inputs

Growing community of inventors

San Francisco, CA, United States of America

Aurko Roy