Naver Corporation

Mert Bulent Sariyildiz

Julien Perez

Philippe Weinzaepfel

Diane Larlus-Larrondo

Diane Larlus

Ioannis Kalantidis

Noé Pion

A training system includes: an encoder module configured to receive a query image and to generate a first vector representative of one or more features in the query image using an encoder; a mixing module configured to generate a second vector by mixing a third vector, representative of one or more features in a second image that is classified as a negative relative to the query image, with a fourth vector; and an adjustment module configured to train the encoder by selectively adjusting one or more parameters of the encoder based on the first vector and the second vector.

Unsupervised contrastive learning of visual representations using negative mixing

A method and system pre-trains a convolutional neural network for image recognition based upon masked language modeling by inputting, to the convolutional neural network, an image; outputting, from the convolutional neural network, a visual embedding tensor of visual embedding vectors; tokenizing a caption to create a list of tokens, at least one token having visual correspondence to the image received by the convolutional neural network; randomly selecting one of the tokens in the list of tokens to be masked, the selected token being taken as ground truth; computing, using a language model neural network, hidden representations of the tokens; using the hidden representation of the masked token, as a query vector, to pool the visual embedding vectors in the visual embedding tensor, attentively; predicting the masked token by mapping the pooled visual embedding vectors to the tokens; determining a prediction loss associated with the masked token; and back-propagating the prediction loss to the convolutional neural network to tune parameters thereof.

Method for training a convolutional neural network for image recognition using image-conditioned masked language modeling

Growing community of inventors

Meylan, France

Mert Bulent Sariyildiz