Google Inc.

Rohan Anil

Aurko Roy

Yonghui Wu

Gregory Sean Corrado

Hrishikesh Balkrishna Aradhye

Andrew M Dai

Tushar Deepak Chandra

Emanuel Taropa

Tal Shaked

Wei Chai

Jeremiah Harmsen

Melvin Jose Johnson Premkumar

Glen Anderson

Orhan Firat

Mustafa Ispir

Ryan P Doherty

Slav Petrov

Yanping Huang

Yuanzhong Xu

David Richard So

Nan Du

Maxim Krikun

Xiaobing Liu

Heng-Tze Cheng

Mehmet Levent Koc

Xavier Eduardo Garcia

Gaurav Mishra

Dmitry Lepikhin

Siamak Shakeri

Yong Cheng

Jonathan Hudson Clark

Erica Ann Moreira

Yujing Zhang

Zachary Alexander Nado

Kefan Xiao

Jin Young Sohn

Eric Jun Jie Ni

Vladimir Feinberg

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes a neural network configured to perform the machine learning task, the neural network comprising an N-grammer layer and an output neural network, the N-grammer layer configured to: at each of one or more heads: receive a sequence of input embeddings; generate a discrete latent representation of the sequence of input embeddings by using a learned product quantization codebook; generate a plurality of n-gram indices from the discrete latent representation; and generate a latent n-gram representation of the sequence of input embeddings; and generate a sequence of output embeddings, and the output neural network configured to: receive the sequence of output embeddings; and process the sequence of output embeddings to generate the network output.

Attention neural networks with N-grammer layers

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform any one or more of a variety of machine learning tasks. For example, the neural network can be configured as a generative neural network, e.g., an autoregressive generative neural network.

Training of large neural networks

A system includes one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the computers to implement a combined machine learning model for processing an input including multiple features to generate a predicted output for the machine learning input. The combined model includes: a deep machine learning model configured to process the features to generate a deep model output; a wide machine learning model configured to process the features to generate a wide model output; and a combining layer configured to process the deep model output generated by the deep machine learning model and the wide model output generated by the wide machine learning model to generate the predicted output, in which the deep model and the wide model have been trained jointly on training data to generate the deep model output and the wide model output.

Wide and deep machine learning models

Growing community of inventors

San Francisco, CA, United States of America

Rohan Anil