Google Inc.

Simon Kornblith

Geoffrey Everest Hinton

Mohammad Norouzi

Ting Chen

Gamaleldin Elsayed

Quoc V Le

Kevin Jordan Swersky

Prajit Ramachandran

Jon Shlens

A computing system can include one or more non-transitory computer-readable media that collectively store a neural network including one or more layers with relaxed spatial invariance. Each of the one or more layers can be configured to receive a respective layer input. Each of the one or more layers can be configured to convolve a plurality of different kernels against the respective layer input to generate a plurality of intermediate outputs, each of the plurality of intermediate outputs having a plurality of portions. Each of the one or more layers can be configured to apply, for each of the plurality of intermediate outputs, a respective plurality of weights respectively associated with the plurality of portions to generate a respective weighted output. Each of the one or more layers can be configured to generate a respective layer output based on the weighted outputs.

Neural network layers with a controlled degree of spatial invariance

Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning. For example, computer-implemented method may include performing semi-supervised contrastive learning based on a set of one or more unlabeled training data, generating an image classification model based on a portion of a plurality of layers in a projection head neural network used in performing the contrastive learning, performing fine-tuning of the image classification model based on a set of one or more labeled training data, and after performing the fine-tuning, distilling the image classification model to a student model comprising a relatively smaller number of parameters than the image classification model.

Systems and methods for contrastive learning of visual representations

Generally, the present disclosure is directed to novel machine-learned classification models that operate with hard attention to make discrete attention actions. The present disclosure also provides a self-supervised pre-training procedure that initializes the model to a state with more frequent rewards. Given only the ground truth classification labels for a set of training inputs (e.g., images), the proposed models are able to learn a policy over discrete attention locations that identifies certain portions of the input (e.g., patches of the images) that are relevant to the classification. In such fashion, the models are able to provide high accuracy classifications while also providing an explicit and interpretable basis for the decision.

Accurate and interpretable classification with hard attention

Provided are systems and methods for contrastive learning of visual representations. In particular, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. In contrast to certain existing techniques, the contrastive self-supervised learning algorithms described herein do not require specialized architectures or a memory bank. Some example implementations of the proposed approaches can be referred to as a simple framework for contrastive learning of representations or 'SimCLR.' Further example aspects are described below and provide the following benefits and insights.

Growing community of inventors

Toronto, Canada

Simon Kornblith