Google Inc.

Maxim Krikun

Yonghui Wu

Melvin Jose Johnson Premkumar

Quoc V Le

Zhifeng Chen

Thorsten Brants

Michael Schuster

Orhan Firat

Yanping Huang

Andrew M Dai

Thang Minh Luong

Ankur Bapna

Slav Petrov

Yuanzhong Xu

David Richard So

Aurko Roy

Nan Du

Rohan Anil

Gaurav Mishra

Xavier Eduardo Garcia

Dmitry Lepikhin

Sneha Kudugunta

Siamak Shakeri

Yong Cheng

Jonathan Hudson Clark

Erica Ann Moreira

Yujing Zhang

Zachary Alexander Nado

Kefan Xiao

Jin Young Sohn

Eric Jun Jie Ni

Vladimir Feinberg

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform any one or more of a variety of machine learning tasks. For example, the neural network can be configured as a generative neural network, e.g., an autoregressive generative neural network.

Training of large neural networks

Systems and methods for routing in mixture-of-expert models. In some aspects of the technology, a transformer may have at least one Mixture-of-Experts ('MoE') layer in each of its encoder and decoder, with the at least one MoE layer of the encoder having a learned gating function configured to route each token of a task to two or more selected expert feed-forward networks, and the at least one MoE layer of the decoder having a learned gating function configured to route each task to two or more selected expert feed-forward networks.

Systems and methods for routing within multitask mixture-of-experts models

Methods, systems, and apparatus, including computer programs encoded on computer storage media for performing machine learning tasks. One method includes receiving (i) a model input, and (ii) data identifying a first machine learning task to be performed on the model input to generate a first type of model output for the model input; augmenting the model input with an identifier for the first machine learning task to generate an augmented model input; and processing the augmented model input using a machine learning model, wherein the machine learning model has been trained on training data to perform a plurality of machine learning tasks including the first machine learning task, and wherein the machine learning model has been configured through training to process the augmented model input to generate a machine learning model output of the first type for the model input.

Implicit bridging of machine learning tasks

Methods, systems, and apparatus, including computer programs encoded on computer storage media for performing machine learning tasks. One method includes receiving (i) a model input, and (ii) data identifying a first machine learning task to be performed on the model input to generate a first type of model output for the model input; augmenting the model input with an identifier for the first machine learning task to generate an augmented model input; and processing the augmented model input using a machine learning model. An exemplary system applying implicit bridging for machine learning tasks, as described in this specification, trains a machine learning model to perform certain types of machine learning tasks without requiring explicit training data for the certain types of machine learning tasks to be used during training.

Growing community of inventors

Castro Valley, CA, United States of America

Maxim Krikun