Google Inc.

Chun-Ta Lu

Zhen Li

Aleksei Timofeev

Andrew Stephen Tomkins

Sujith Ravi

Thomas J Duerig

Futang Peng

Da-Cheng Juan

Yaxi Gao

Yi-ting Chen

Ariel Fuxman

Chen Sun

Manan Shah

Chao Jia

Krishnamurthy Viswanathan

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an image embedding model. In one aspect, a method comprises: obtaining training data comprising a plurality of training examples, wherein each training example comprises: an image pair comprising a first image and a second image; and selection data indicating one or more of: (i) a co-click rate of the image pair, and (ii) a similar-image click rate of the image pair; and using the training data to train an image embedding model having a plurality of image embedding model parameters.

Training image and text embedding models

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for realizing a multimodal image classifier. In an aspect, a method includes, for each image of a plurality of images: processing the image by a textual generator model to obtain a set of phrases that are descriptive of the content of the image, wherein each phrase is one or more terms, processing the set of phrases by a textual embedding model to obtain an embedding of predicted text for the image, and processing the image using an image embedding model to obtain an embedding of image pixels of the image. Then a multimodal image classifier is trained on the embeddings of predicted text for the images and the embeddings of image pixels for the images to produce, as output, labels of an output taxonomy to classify an image based on the image as input.

Multimodal image classifier using textual and visual embeddings

Growing community of inventors

Sunnyvale, CA, United States of America

Chun-Ta Lu