Google Inc.

Baidu USA LLC

Zihang Dai

Quoc V Le

Hanxiao Liu

Noam M Shazeer

Mingxing Tan

David Richard So

Wojciech Andrzej Manke

Wei Yu

Thang Minh Luong

Wei Xu

Golnaz Ghiasi

Hieu Hy Pham

Qizhe Xie

Lei Li

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using memory-optimized contrastive learning to train image encoder and text encoder neural networks.

Memory-optimized contrastive learning

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.

Granular neural network architecture search over low-level primitives

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a IT machine learning model. One of the methods includes receiving training data comprising a plurality of unlabeled training inputs and a plurality of labeled training inputs; generating augmented training data, comprising generating, for each of the plurality of unlabeled training inputs, a respective augmented training input by applying a data augmentation technique to the unlabeled training input; and training the machine learning model on the augmented training data. In particular, but not exclusively, the model may be trained for perceptual tasks (e.g. tasks relating to vision or speech).

Training machine learning models using unsupervised data augmentation

A computer-implemented method for performing computer vision with reduced computational cost and improved accuracy can include obtaining, by a computing system including one or more computing devices, input data comprising an input tensor having one or more dimensions, providing, by the computing system, the input data to a machine-learned convolutional attention network, the machine-learned convolutional attention network including two or more network stages, and, in response to providing the input data to the machine-learned convolutional attention network, receiving, by the computing system, a machine-learning prediction from the machine-learned convolutional attention network. The convolutional attention network can include at least one attention block, wherein the attention block includes a relative attention mechanism, the relative attention mechanism including the sum of a static convolution kernel with an adaptive attention matrix. This provides for improved generalization, capacity, and efficiency of the convolutional attention network relative to some existing models.

Systems and methods for machine-learned models having convolution and attention

Described herein are systems and methods for determining how to automatically answer questions like 'Where did Harry Potter go to school?' Carefully built knowledge graphs provide rich sources of facts. However, it still remains a challenge to answer factual questions in natural language due to the tremendous variety of ways a question can be raised. Presented herein are embodiments of systems and methods for human inspired simple question answering (HISQA), a deep-neural-network-based methodology for automatic question answering using a knowledge graph. Inspired by human's natural actions in this task, embodiments first find the correct entity via entity linking, and then seek a proper relation to answer the question&#x2014;both achieved by deep gated recurrent networks and neural embedding mechanism.

Systems and methods for human inspired simple question answering (HISQA)

Growing community of inventors

Pittsburgh, PA, United States of America

Zihang Dai