Google Inc.

David Richard So

Quoc V Le

Noam M Shazeer

Hanxiao Liu

Zihang Dai

Wojciech Andrzej Manke

Yonghui Wu

Andrew M Dai

Melvin Jose Johnson Premkumar

Orhan Firat

Slav Petrov

Yanping Huang

Yuanzhong Xu

Aurko Roy

Maxim Krikun

Nan Du

David Martin Dohan

Rohan Anil

Gaurav Mishra

Chen Liang

Xavier Eduardo Garcia

Dmitry Lepikhin

Siamak Shakeri

Yong Cheng

Jonathan Hudson Clark

Erica Ann Moreira

Yujing Zhang

Zachary Alexander Nado

Kefan Xiao

Jin Young Sohn

Eric Jun Jie Ni

Vladimir Feinberg

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.

Granular neural network architecture search over low-level primitives

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform any one or more of a variety of machine learning tasks. For example, the neural network can be configured as a generative neural network, e.g., an autoregressive generative neural network.

Training of large neural networks

A method for receiving training data for training a neural network to perform a machine learning task and for searching for, using the training data, an optimized neural network architecture for performing the machine learning task is described. Searching for the optimized neural network architecture includes: maintaining population data; maintaining threshold data; and repeatedly performing the following operations: selecting one or more candidate architectures from the population data; generating a new architecture from the one or more selected candidate architectures; for the new architecture: training a neural network having the new architecture until termination criteria for the training are satisfied; and determining a final measure of fitness of the neural network having the new architecture after the training; and adding data defining the new architecture and the final measure of fitness for the neural network having the new architecture to the population data.

Computationally efficient neural network architecture search

Growing community of inventors

Brooklyn, NY, United States of America

David Richard So