Google Inc.

Thomas J Duerig

Charles Joseph Rosenberg

Samy Bengio

Kunlong Gu

Zhen Hao Zhou

Neil G Alldrin

Andrew Stephen Tomkins

Sujith Ravi

Yangli Hector Yee

Arcot J Preetham

Zhen Li

Futang Peng

Da-Cheng Juan

Aleksei Timofeev

Chun-Ta Lu

Yi-ting Chen

David Cai

Yaxi Gao

Jason Edward Weston

Erik Murphy-Chutorian

Yun Zhou

Maya Rani Gupta

Maks Ovsjanikovs

Sean Arietta

Mingchen Gao

Scott Alexander Rudkin

Hongsheng Wang

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an image embedding model. In one aspect, a method comprises: obtaining training data comprising a plurality of training examples, wherein each training example comprises: an image pair comprising a first image and a second image; and selection data indicating one or more of: (i) a co-click rate of the image pair, and (ii) a similar-image click rate of the image pair; and using the training data to train an image embedding model having a plurality of image embedding model parameters.

Training image and text embedding models

The present disclosure is directed to methods and systems for knowledge distillation. Implementations of the disclosure can include executing the following actions using one or more computing devices: obtaining an initial training dataset including multiple training examples; determining sets of outputs by performing inference on the training examples with a group of pre-trained machine-learned models that have been trained to perform a respective task based on a respective pre-trained model training dataset; evaluating a performance of each pretrained machine-learned model based at least in part on the set of outputs generated by the pre-trained machine-learned model; determining for the set of outputs generated by each pre-trained machine-learned model, whether to include one or more outputs of the set of outputs in a distillation training dataset based at least in part on the respective performance of such pre-trained machine-learned model; and training a distilled machine-learned model using the distillation training dataset.

Systems and methods for performing knowledge distillation

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for analyzing image search result relevance. In one aspect, a method includes receiving result data specifying a search query and responsive image search results that reference images that are responsive to the search query. A determination is made that the search query matches an indexed query. An image relevance model is identified for the indexed query. The image relevance model can output a relevance score adjustment factor for an image search result based on image feature values of the image that is referenced by the search result. A relevance score adjustment factor is determined for each image search result using the identified image relevance model. A relevance score for each image search result is adjusted using the image's image relevance score adjustment factor. The images are ranked based on the adjusted relevance scores.

Online image analysis

Implementations include actions of receiving an initial data set including a plurality of images, each image being associated with a set of labels, wherein each label in the set of labels is assigned to a respective image of the plurality of images by an initial model, the initial model being specific to a ground-truth label; for each image in the plurality of images: providing a list of categories associated with a respective image based on a respective set of labels, and determining a primary category of the respective image based on the list of categories; determining a category of the ground-truth label; and providing a revised data set based on the initial data set by comparing the category to primary categories of respective images in the plurality of images, the initial model being trained based on the revised data set to provide a revised model.

Image annotation based on label consensus

Methods, systems and apparatus for refining image relevance models. In general, one aspect includes receiving a trained image relevance model that generates relevance measures of content feature values of images to a query, identifying a first threshold number of common content feature values for the set of training images, the common content feature values being identified as a set of content feature values that are each shared by at least a portion of the training images, identifying a subset of the set of training images having a quantity of the common content feature values greater than a second threshold number of content features values, and generating a re-trained image relevance model based on content feature values of the set of training images, wherein content feature values of the subset of training images are weighted higher than content feature values of the training images not in the subset.

Refining image relevance models

Methods, systems and apparatus for refining image relevance models. In general, one aspect of the subject matter described in this specification can be implemented in methods that include re-training an image relevance model by generating a first re-trained model based on content feature values of first images of a first portion of training images in a set of training images, receiving, from the first re-trained model, image relevance scores for second images of a second portion of the set of training images, removing, from the set of training images, some of the second images identified as outlier images for which the image relevance score received from the first re-trained model is below a threshold score, and generating a second re-trained model based on content feature values of the first images of the first portion and the second images of the second portion that remain following removal of the outlier images.

A method includes receiving a search query comprising one or more query terms, receiving image relevance models that each generate relevance measures of content feature values of images to a query, each image relevance model being a predictive model that has been trained based on content feature values of a set of training images, and each of the queries being a unique set of one or more query terms received by a search system as a query input, identifying an image relevance model for a different query that has been identified as similar to the received search query, and calculating a fractional adjustment multiplier for search results responsive to the received search query, the fractional adjustment multiplier being based on a relevance measure generated by the identified image relevance model for the different query and based on a degree of similarity between the different query and the received search query.

Applying query based image relevance models

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying images responsive to a search phrase are disclosed. In one aspect, a method includes identifying a set of responsive images for a search phrase that includes two or more terms. Interaction rankings are determined for images in the set of responsive images. Two or more sub-queries are created based on the search phrase. Sub-query model rankings are determined for images in the set of responsive images. A search phrase score is determined for the image relevance model. Based on the search phrase scores for the sub-queries, one of the sub-query models is selected as a model for the search phrase.

Sub-query evaluation for image search

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for adjusting confidence scores of image labels for images. In one aspect, a method includes accessing images stored in an image data store, the images being associated with respective sets of labels, the labels describing content depicted in the image and having a respective confidence score that is a measure of confidence that the label accurately describes the content depicted in the image; selecting a first image from the images and determining for each of the other images and independent of the labels, a proximity score that is a measure of a relatedness of the other image to the first image; and adjusting the set of labels associated with the first image based on the respective proximity scores of the other images and the confidence scores of the labels of the other images.

Semantic image label synthesis

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for classifying images. In one aspect, a method includes receiving training samples for a particular data dimension. Each training sample specifies a training value for the data dimension and a measure of relevance between the training sample and a phrase. A value range is determined for the data dimension. The value range is segmented into two or more segments. A predictive model is trained for each segment. The predictive model for each segment is trained to predict an output based on an input value that is within the segment. A classification sample specifying an input value is received. A classification output is computed based on the input value, the predictive model for the segment in which the input value is included, and the predictive model for an adjacent segment.

Image classification

Methods, systems and apparatus for assigning labels to images. In general, in one aspect, a method includes determining, for an image, a first set of labels, each label being determined to be indicative of subject matter of the image based on content feature values of the image, for each label in the first set of labels, determining a second set of labels, each label in the second set of labels determined to be semantically related to the label in the first set of labels, assigning a score to each label in the second sets of labels, and based on the scores assigned to each label in the second sets of labels, assigning one or more of the labels in the second sets of labels to the image.

Growing community of inventors

Sunnyvale, CA, United States of America

Thomas J Duerig