Google Inc.

Quoc V Le

Barret Zoph

Jeffrey Adgate Dean

Navdeep Jaitly

Oriol Vinyals

Vijay Vasudevan

Mingxing Tan

Ekin Dogus Cubuk

Ilya Sutskever

Jonathon Shlens

Andrew M Dai

Samuel Bengio

Noam M Shazeer

Azalia Mirhoseini

Thang Minh Luong

Yonghui Wu

Zhifeng Chen

William Chan

Hieu Hy Pham

Anna Darling Goldie

Gregory Sean Corrado

Kevin Stefan Clark

Golnaz Ghiasi

Zihang Dai

Brian Strope

Ron J Weiss

Ruoming Pang

Ioannis Agiomyrgiannakis

Ryan Michael Rifkin

Chian-Min Richard Ho

Wei Yu

Mohammad Norouzi

James Laudon

Robert Andrew James Clark

Wenjie Jiang

Russell J Ryan

Mustafa Nazim Yazgan

Tsung-Yi Lin

Yuxuan Wang

Benoit Steiner

Shen Wang

Zongheng Yang

Hanxiao Liu

Roger David Carpenter

Ebrahim Songhori

Young-Joon Lee

Alok Aggarwal

Bo Chen

Michael Schuster

Yanping Huang

Norman Paul Jouppi

Naveen Kumar

Kai Chen

Yi Zhang

Yu Zhang

Kun Zhang

Samy Bengio

James Wilson

Thorsten Brants

Yingwei Cui

Sheng Li

Chung-Cheng Chiu

Liqun Cheng

Nissan Hajaj

Yoram Singer

Dengyong Zhou

Alvin Rishi Rajkomar

Melvin Jose Johnson Premkumar

Peter Junteng Liu

Tomas Mikolov

Rasmus Munk Larsen

Pranav Khaitan

Cong Li

Hongrae Lee

Eyal Oren

Benjamin Irvine

Wojciech Zaremba

James Wexler

Martin Guy Dixon

Parthasarathy Ranganathan

Sridhar Lakshmanamurthy

Sergey Levine

Geoffrey Everest Hinton

Matthias Heiler

Yu Shrike Zhang

Martin Baeuml

Jiquan Ngiam

Xiaodan Song

Daniel Stodolsky

Aleksandra Faust

Honglak Lee

Thushan Amarasiriwardena

Vikram Sridar

Daniel De Freitas Adiwardana

Andrew Li

Yifeng Lu

Simon Kornblith

David J Bieber

Malte Nuhn

A method for searching for an output machine learning (ML) algorithm to perform an ML task is described. The method comprising: receiving data specifying an input ML algorithm; receiving data specifying a search algorithm that searches for candidate ML algorithms and an evaluation function that evaluates the performance of candidate ML algorithms; generating data representing a symbolic tree from the input ML algorithm; generating data representing a hyper symbolic tree from the symbolic tree; searching an algorithm search space that defines a set of possible concrete symbolic trees from the hyper symbolic tree for candidate ML algorithms and training the candidate ML algorithms to determine a respective performance metric for each candidate ML algorithm; and selecting one or more trained candidate ML algorithms among the trained candidate ML algorithms based on the determined performance metrics.

Machine learning algorithm search with symbolic programming

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for generating and searching reinforcement learning algorithms. In some implementations, a computer-implemented system generates a sequence of candidate reinforcement learning algorithms. Each candidate reinforcement learning algorithm in the sequence is configured to receive an input environment state characterizing a state of an environment and to generate an output that specifies an action to be performed by an agent interacting with the environment. For each candidate reinforcement learning algorithm in the sequence, the system performs a performance evaluation for a set of a plurality of training environments. For each training environment, the system adjusts a set of environment-specific parameters of the candidate reinforcement learning algorithm by performing training of the candidate reinforcement learning algorithm to control a corresponding agent in the training environment. The system generates an environment-specific performance metric for the candidate reinforcement learning algorithm that measures a performance of the candidate reinforcement learning algorithm in controlling the corresponding agent in the training environment as a result of the training. After performing training in the set of training environments, the system generates a summary performance metric for the candidate reinforcement learning algorithm by combining the environment-specific performance metrics generated for the set of training environments. After evaluating each of the candidate reinforcement learning algorithms in the sequence, the system selects one or more output reinforcement learning algorithms from the sequence based on the summary performance metrics of the candidate reinforcement learning algorithms.

Reinforcement learning algorithm search

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining update rules for training neural networks. One of the methods includes generating, using a controller neural network, a batch of output sequences, each output sequence in the batch defining a respective update rule; for each output sequence in the batch: training a respective instance of a child neural network using the update rule defined by the output sequence; evaluating a performance of the trained instance of the child neural network on the particular neural network task to determine a performance metric for the trained instance of the child neural network on the particular neural network task; and using the performance metrics for the trained instances of the child neural network to adjust the current values of the controller parameters of the controller neural network.

Neural network optimizer search

Aspects of the disclosure are directed to a heterogeneous machine learning accelerator system with compute and memory nodes connected by high speed chip-to-chip interconnects. While existing remote/disaggregated memory may require memory expansion via remote processing units, aspects of the disclosure add memory nodes into machine learning accelerator clusters via the chip-to-chip interconnects without needing assistance from remote processing units to achieve higher performance, simpler software stack, and/or lower cost. The memory nodes may support prefetch and intelligent compression to enable the use of low cost memory without performance degradation.

Heterogeneous ML accelerator cluster with flexible system resource balance

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using memory-optimized contrastive learning to train image encoder and text encoder neural networks.

Memory-optimized contrastive learning

A method for receiving training data for training a neural network (NN) to perform a machine learning (ML) task and for determining, using the training data, an optimized NN architecture for performing the ML task is described. Determining the optimized NN architecture includes: maintaining population data comprising, for each candidate architecture in a population of candidate architectures, (i) data defining the candidate architecture, and (ii) data specifying how recently a neural network having the candidate architecture has been trained while determining the optimized neural network architecture; and repeatedly performing multiple operations using each of a plurality of worker computing units to generate a new candidate architecture based on a selected candidate architecture having the best measure of fitness, adding the new candidate architecture to the population, and removing from the population the candidate architecture that was trained least recently.

Regularized neural network architecture search

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.

Granular neural network architecture search over low-level primitives

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes generating, using a controller neural network having controller parameters and in accordance with current values of the controller parameters, a batch of output sequences. The method includes, for each output sequence in the batch: generating an instance of a child convolutional neural network (CNN) that includes multiple instances of a first convolutional cell having an architecture defined by the output sequence; training the instance of the child CNN to perform an image processing task; and evaluating a performance of the trained instance of the child CNN on the task to determine a performance metric for the trained instance of the child CNN; and using the performance metrics for the trained instances of the child CNN to adjust current values of the controller parameters of the controller neural network.

Neural architecture search for convolutional neural networks

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on an input to generate an output. In one aspect, one of the method includes receiving input data that describes an input of a machine learning task; receiving candidate output data that describes a set of candidate classification outputs of the machine learning task for the input; generating an input sequence that includes the input and the set of candidate classification outputs; processing the input sequence using a neural network to generate a network output that specifies a respective score for each candidate classification output in the set of candidate classification outputs; and generating an output of the machine learning task for the input, comprising selecting, as the output, a selected candidate classification output from the set of candidate classification outputs using the respective scores.

Performing machine learning tasks using instruction-tuned neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model. One of the methods includes obtaining a training data set for training a machine learning model, the training data set comprising a plurality of training inputs; determining a plurality of data augmentation policies, wherein each data augmentation policy defines a procedure for processing a training input to generate a transformed training input; for each data augmentation policy, training the machine learning model using the data augmentation policy; determining, for each data augmentation policy, a quality measure of the machine learning model that has been trained using the data augmentation policy; and selecting a final data augmentation policy based using the quality measures of the machine learning models.

Training neural networks using data augmentation policies

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating document vector representations. One of the methods includes obtaining a new document; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system has been trained to receive an input document and a sequence of words from the input document and to generate a respective word score for each word in a set of words, wherein each of the respective word scores represents a predicted likelihood that the corresponding word follows a last word in the sequence in the input document, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of sequences of words to the trained neural network system to determine the vector representation for the new document using gradient descent.

Generating vector representations of documents

An example technique for image analysis is provided. An example image analysis method includes obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response. The example image analysis method includes inputting, to a machine-learned model, the instructive sequence and an operative image processing query that comprises image data, wherein the machine-learned model is configured to process the operative query with attention over the instructive sequence. The example method can include generating, using the machine-learned model and responsive to the operative query, an operative image processing response that comprises an analysis of the image data.

Image analysis by prompting of machine-learned models using chain of thought

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining neural network architectures. One of the methods includes generating, using a controller neural network, a batch of output sequences, each output sequence in the batch specifying a respective subset of a plurality of components of a large neural network that should be active during the processing of inputs by the large neural network; for each output sequence in the batch: determining a performance metric of the large neural network on the particular neural network task (i) in accordance with current values of the large network parameters and (ii) with only the subset of components specified by the output sequences active; and using the performance metrics for the output sequences in the batch to adjust the current values of the controller parameters of the controller neural network.

Neural architecture search

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using a priority queue. One of the methods includes maintaining data identifying a set of K output sequences that were previously generated; selecting at least one of the output sequences from the set of output sequences; for each selected output sequence, determining a respective score; determining, for each selected sequence, a respective first update to the current values of the controller parameters; generating a batch of new output sequences using the controller neural network; obtaining a respective reward for each of the new output sequences; determining, from the new output sequences and the output sequences in the maintained data, the K output sequences that have the highest rewards; and modifying the maintained data.

Training neural networks using priority queues

Growing community of inventors

Stanford, CA, United States of America

Quoc V Le