Google Inc.

Harvard College

University of Toronto

Socpra Sciences Et Genie S.e.c.

Kevin Jordan Swersky

Milad Olia Hashemi

Jamie Alexander Smith

Ryan P Adams

Roland Jasper Snoek

Geoffrey Everest Hinton

Ting Chen

Richard Zemel

Parthasarathy Ranganathan

Mohammad Norouzi

Simon Kornblith

Hugo Larochelle

Simon Komblith

Methods, systems, and apparatus, including computer programs encoded on computer storage media, relating to multi-task recurrent neural networks. One of the methods includes maintaining data specifying, for a recurrent neural network, a separate internal state for each of a plurality of memory regions; receiving a current input; identifying a particular memory region of the memory access address defined by the current input; selecting, from the internal states specified in the maintained data, the internal state for the particular memory region; processing, in accordance with the selected internal state for the particular memory region, the current input in the sequence of inputs using the recurrent neural network to: generate an output, the output defining a probability distribution of a predicted memory access address, and update the selected internal state of the particular memory region; and associating the updated selected internal state with the particular memory region in the maintained data.

Multi-task recurrent neural networks

Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning. For example, computer-implemented method may include performing semi-supervised contrastive learning based on a set of one or more unlabeled training data, generating an image classification model based on a portion of a plurality of layers in a projection head neural network used in performing the contrastive learning, performing fine-tuning of the image classification model based on a set of one or more labeled training data, and after performing the fine-tuning, distilling the image classification model to a student model comprising a relatively smaller number of parameters than the image classification model.

Systems and methods for contrastive learning of visual representations

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for pre-fetching data from memory using neural networks. One example system receives a sequence of prior program counter addresses of a computer program and corresponding delta values. The system creates an input representation based on the sequence. The system provides the input representation as input to a recurrent neural network. The system receives from the recurrent neural network an output that defines a probability distribution over future delta values. Each probability in the distribution represents a likelihood that execution of a future instruction of the computer program will cause data to be fetched from a particular future memory address.

Computer system prediction machine learning models

Techniques for use in connection with performing optimization using an objective function that maps elements in a first domain to values in a range. The techniques include using at least one computer hardware processor to perform: identifying a first point at which to evaluate the objective function at least in part by using an acquisition utility function and a probabilistic model of the objective function, wherein the probabilistic model depends on a non-linear one-to-one mapping of elements in the first domain to elements in a second domain; evaluating the objective function at the identified first point to obtain a corresponding first value of the objective function; and updating the probabilistic model of the objective function using the first value to obtain an updated probabilistic model of the objective function.

Systems and methods for Bayesian optimization using non-linear mapping of input

Techniques for use in connection with performing optimization using a plurality of objective functions associated with a respective plurality of tasks. The techniques include using at least one computer hardware processor to perform: identifying, based at least in part on a joint probabilistic model of the plurality of objective functions, a first point at which to evaluate an objective function in the plurality of objective functions; selecting, based at least in part on the joint probabilistic model, a first objective function in the plurality of objective functions to evaluate at the identified first point; evaluating the first objective function at the identified first point; and updating the joint probabilistic model based on results of the evaluation to obtain an updated joint probabilistic model.

Systems and methods for multi-task Bayesian optimization

Growing community of inventors

Toronto, Canada

Kevin Jordan Swersky