Google Inc.

Microsoft Technology Licensing, LLC

Geoffrey Everest Hinton

Ilya Sutskever

Alexander Krizhevsky

Jeffrey Adgate Dean

Oriol Vinyals

Ting Chen

Mohammad Norouzi

Sara Sabour Rouh Aghdam

Simon Kornblith

Nicholas Myles Wisener Frosst

Nitish Srivastava

Navdeep Jaitly

William Chan

Sofien Bouaziz

Kevin Jordan Swersky

Kyle Genova

Andrea Tagliasacchi

Boyang Deng

Soroosh Yazdani

Quoc V Le

Volodymyr Mnih

Christopher Michael Bishop

Chitwan Saharia

Yee Whye Teh

Jamie Ryan Kiros

Nitish Srivastva

Johan Fredrik Markus Svensen

Simon Komblith

Adam Roman Kosiorek

Nicolas Guy Robert Papernot

Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning. For example, computer-implemented method may include performing semi-supervised contrastive learning based on a set of one or more unlabeled training data, generating an image classification model based on a portion of a plurality of layers in a projection head neural network used in performing the contrastive learning, performing fine-tuning of the image classification model based on a set of one or more labeled training data, and after performing the fine-tuning, distilling the image classification model to a student model comprising a relatively smaller number of parameters than the image classification model.

Systems and methods for contrastive learning of visual representations

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sequence modeling. One of the methods includes receiving an input sequence having a plurality of input positions; determining a plurality of blocks of consecutive input positions; processing the input sequence using a neural network to generate a latent alignment, comprising, at each of a plurality of input time steps: receiving a partial latent alignment from a previous input time step; selecting an input position in each block, wherein the token at the selected input position of the partial latent alignment in each block is a mask token; and processing the partial latent alignment and the input sequence using the neural network to generate a new latent alignment, wherein the new latent alignment comprises, at the selected input position in each block, an output token or a blank token; and generating, using the latent alignment, an output sequence.

Sequence modeling using imputation

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for detecting objects in images. One of the methods includes obtaining an input image; processing the input image to generate predicted part feature data, the predicted part feature data comprising, for each of a plurality of possible object parts: a part presence probability representing a likelihood that the possible object part is depicted in the input image, a predicted pose of the possible object part in the input image given that the possible object part is depicted in the input image, and an object part feature vector characterizing the depiction of the possible object part given that the possible object part is depicted in the input image; and processing the predicted part feature data for the plurality of possible object parts to generate an object detection output that identifies one or more objects depicted in the input image.

Object discovery in images through categorizing object parts

Methods, systems, and apparatus including computer programs encoded on a computer storage medium, for generating convex decomposition of objects using neural network models. One of the methods includes receiving an input that depicts an object. The input is processed using a neural network to generate an output that defines a convex representation of the object. The output includes, for each of a plurality of convex elements, respective parameters that define a position of the convex element in the convex representation of the object.

Convex representation of objects using neural network

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a classification neural network. In one aspect, a method comprises: for each of a plurality of network inputs: processing the network input using the classification neural network to generate a classification output that defines a predicted class of the network input; determining a soft nearest neighbor loss, wherein the soft nearest neighbor loss encourages intermediate representations of network inputs of different classes to become more entangled, wherein the entanglement of intermediate representations of network inputs of different classes characterizes how similar pairs of intermediate representations of network inputs of different class are relative to pairs of intermediate representations of network inputs of the same class; and adjusting the current values of the classification neural network parameters using gradients of the soft nearest neighbor loss with respect to the classification neural network parameters.

Neural network training using the soft nearest neighbor loss

A parallel convolutional neural network is provided. The CNN is implemented by a plurality of convolutional neural networks each on a respective processing node. Each CNN has a plurality of layers. A subset of the layers are interconnected between processing nodes such that activations are fed forward across nodes. The remaining subset is not so interconnected.

System and method for parallelizing convolutional neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a distilled machine learning model. One of the methods includes training a cumbersome machine learning model, wherein the cumbersome machine learning model is configured to receive an input and generate a respective score for each of a plurality of classes; and training a distilled machine learning model on a plurality of training inputs, wherein the distilled machine learning model is also configured to receive inputs and generate scores for the plurality of classes, comprising: processing each training input using the cumbersome machine learning model to generate a cumbersome target soft output for the training input; and training the distilled machine learning model to, for each of the training inputs, generate a soft output that matches the cumbersome target soft output for the training input.

Training distilled machine learning models

A system for training a neural network. A switch is linked to feature detectors in at least some of the layers of the neural network. For each training case, the switch randomly selectively disables each of the feature detectors in accordance with a preconfigured probability. The weights from each training case are then normalized for applying the neural network to test data.

System and method for addressing overfitting in a neural network

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for a neural network that is configured to receive a network input and to generate a network output for the network input. The neural network comprises a plurality of layers arranged in a sequence, including a plurality of capsule layers. Each particular capsule in a particular capsule layer is configured to receive respective inputs including: (i) outputs generated by capsules of a previous capsule layer that is before the particular capsule layer in the sequence, and (ii) final routing factors between capsules of the previous capsule layer and the particular capsule, wherein the final routing factors are generated by a routing subsystem. Each particular capsule in the particular capsule layer is configured to determine a particular capsule output based on the received inputs, wherein the particular capsule output is of dimension greater than one.

Capsule neural networks

Provided are systems and methods for contrastive learning of visual representations. In particular, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. In contrast to certain existing techniques, the contrastive self-supervised learning algorithms described herein do not require specialized architectures or a memory bank. Some example implementations of the proposed approaches can be referred to as a simple framework for contrastive learning of representations or 'SimCLR.' Further example aspects are described below and provide the following benefits and insights.

Growing community of inventors

Toronto, Canada

Geoffrey Everest Hinton