Google Inc.

Yanping Huang

Quoc V Le

Alok Aggarwal

Esteban Alberto Real

Orhan Firat

Maxim Krikun

Yonghui Wu

Zhifeng Chen

Andrew M Dai

Jiquan Ngiam

Thang Minh Luong

Melvin Jose Johnson Premkumar

Ankur Bapna

Slav Petrov

Dehao Chen

Yuanzhong Xu

Aurko Roy

David Richard So

Nan Du

Rohan Anil

Gaurav Mishra

Xavier Eduardo Garcia

Jin Young Sohn

Eric Jun Jie Ni

Vladimir Feinberg

HyoukJoong Lee

Dmitry Lepikhin

Kefan Xiao

Sneha Kudugunta

Youlong Cheng

Siamak Shakeri

Yong Cheng

Jonathan Hudson Clark

Erica Ann Moreira

Yujing Zhang

Zachary Alexander Nado

A method for receiving training data for training a neural network (NN) to perform a machine learning (ML) task and for determining, using the training data, an optimized NN architecture for performing the ML task is described. Determining the optimized NN architecture includes: maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) data specifying how recently a neural network having the candidate architecture has been trained while determining the optimized neural network architecture; and repeatedly performing multiple operations using each of a plurality of worker computing units to generate a new candidate architecture based on a selected candidate architecture having the best measure of fitness, adding the new candidate architecture to the population, and removing from the population the candidate architecture that was trained least recently.

Regularized neural network architecture search

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform any one or more of a variety of machine learning tasks. For example, the neural network can be configured as a generative neural network, e.g., an autoregressive generative neural network.

Training of large neural networks

Systems and methods for routing in mixture-of-expert models. In some aspects of the technology, a transformer may have at least one Mixture-of-Experts ('MoE') layer in each of its encoder and decoder, with the at least one MoE layer of the encoder having a learned gating function configured to route each token of a task to two or more selected expert feed-forward networks, and the at least one MoE layer of the decoder having a learned gating function configured to route each task to two or more selected expert feed-forward networks.

Systems and methods for routing within multitask mixture-of-experts models

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training giant neural networks. One of the methods includes obtaining data specifying a partitioning of the neural network into N composite layers that form a sequence of composite layers, wherein each composite layer comprises a distinct plurality of layers from the multiple network layers of the neural network; obtaining data assigning each of the N composite layers to one or more computing devices from a set of N computing devices; partitioning a mini-batch of training examples into a plurality of micro-batches; and training the neural network, comprising: performing a forward pass through the neural network until output activations have been computed for each micro-batch for a final composite layer in the sequence, and performing a backward pass through the neural network until output gradients have been computed for each micro-batch for the first composite layer in the sequence.

Training giant neural networks using pipeline parallelism

Growing community of inventors

Mountain View, CA, United States of America

Yanping Huang