Google Inc.

Deepmind Technologies Limited

Hanxiao Liu

Quoc V Le

Noam M Shazeer

Mingxing Tan

Wojciech Andrzej Manke

Oriol Vinyals

Karen Simonyan

Koray Kavukcuoglu

Azalia Mirhoseini

Zihang Dai

David Richard So

James Laudon

Wei Yu

Anna Darling Goldie

Golnaz Ghiasi

Shen Wang

Hieu Hy Pham

Sudip Roy

Qiumin Xu

Amirali Abdolrashidi

Yanqi Zhou

Daniel Lin-Kit Wong

Chrisantha Thomas Fernando

Chao Ma

Phitchaya Mangpo Phothilimthana

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for compiler optimizations using a compiler optimization network. One of the methods includes receiving an input program, wherein the input program defines a graph of operation modules, wherein each node in the graph is a respective operation module, and each edge between nodes in the graph represents one operation module receiving the output generated by another operation module. The input program is processed by a compiler optimization network comprising a graph-embedding network that is configured to encode operation features and operation dependencies of the operation modules of the input program into a graph embedding representation and a policy network that is configured to generate an optimization action for each of one or more nodes encoded in the graph embedding representation. The compiler optimization network generates an output optimization plan comprising one or more optimization actions for the input program.

Learned graph optimizations for compilers

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using memory-optimized contrastive learning to train image encoder and text encoder neural networks.

Memory-optimized contrastive learning

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.

Granular neural network architecture search over low-level primitives

A computer-implemented method for automatically determining a neural network architecture represents a neural network architecture as a data structure defining a hierarchical set of directed acyclic graphs in multiple levels. Each graph has an input, an output, and a plurality of nodes between the input and the output. At each level, a corresponding set of the nodes are connected pairwise by directed edges which indicate operations performed on outputs of one node to generate an input to another node. Each level is associated with a corresponding set of operations. At a lowest level, the operations associated with each edge are selected from a set of primitive operations. The method includes repeatedly generating new sample neural network architectures, and evaluating their fitness. The modification is performed by selecting a level, selecting two nodes at that level, and modifying, removing or adding an edge between those nodes according to operations associated with lower levels of the hierarchy.

Using hierarchical representations for neural network architecture searching

A computer-implemented method for performing computer vision with reduced computational cost and improved accuracy can include obtaining, by a computing system including one or more computing devices, input data comprising an input tensor having one or more dimensions, providing, by the computing system, the input data to a machine-learned convolutional attention network, the machine-learned convolutional attention network including two or more network stages, and, in response to providing the input data to the machine-learned convolutional attention network, receiving, by the computing system, a machine-learning prediction from the machine-learned convolutional attention network. The convolutional attention network can include at least one attention block, wherein the attention block includes a relative attention mechanism, the relative attention mechanism including the sum of a static convolution kernel with an adaptive attention matrix. This provides for improved generalization, capacity, and efficiency of the convolutional attention network relative to some existing models.

Systems and methods for machine-learned models having convolution and attention

Growing community of inventors

Santa Clara, CA, United States of America

Hanxiao Liu