Google Inc.

Pinterest, Inc.

Charles Joseph Rosenberg

Samy Bengio

Erik Murphy-Chutorian

Jingbin Wang

Henry Allan Rowley

Yangli Hector Yee

Radhika Malpani

Thomas J Duerig

Yang Song

Kunlong Gu

Hartwig Adam

Andrew Huan Zhai

Anelia Angelova

Sean O'Malley

Sarah Moussa

Hartmut Neven

David Petrou

Sergey Ioffe

Yuan Hang Li

Jiang Wang

Gabriel Taubman

Louis Wang

Ulrich Franz Buddemeier

Fernando A Brucher

Zhen Hao Zhou

Charles Yang

Neil G Alldrin

Maks Ovsjanikov

Nikil Pancha

Michele Mae Covell

Yushi Jing

Hui Li

Ting Liu

Andrea Lynn Frome

Jurij Leskovec

Yoram Singer

Bo Chen

Chantat Eksombatchai

Bo Zhao

Thomas Leung

Arcot J Preetham

Aditya Pal

Kaifeng Chen

John Zhang

Nemanja Petrovic

Yitong Zhou

Xiangrong Chen

Andrew Yan-Tak Ng

Bin Shen

Mei Han

Jason Edward Weston

Shengyang Dai

Congcong Li

Kyle Howard Heath

Gabriel Wolosin

Ehud Rivlin

Yun Zhou

Eric Kim

Maya Rani Gupta

Zhongli Ding

Henrik C Stewenius

Mohamed Aly

Chenyi Li

Maks Ovsjanikovs

Sean Arietta

Mingchen Gao

Sean O Malley

Described are systems and methods for determining complementary and/or matching objects based on an input query object. The described systems and methods can generate an embedding representative of the provided object, which can be transformed to generate a style embedding by a trained system, such as a machine learning system. The style embedding can then be used to identify one or more complementary objects from a corpus of classified objects. Aspects of the present disclosure also relate to creation of the training dataset, as well as training the machine learning system.

Systems and methods for identifying complementary objects having similar styles

Systems and methods for identifying content for an input query are presented. A mapping model is trained to map elements of an input query embedding vector for a received query into one or more elements of a destination embedding vector. In response to receiving an input query, an input query embedding vector is generated that projects into an input query embedding space. The input query embedding vector is processed by the mapping model to map the input query embedding vector into one or more elements of a destination embedding vector in a destination embedding space, resulting in a partial destination embedding vector. Items of a corpus of content are projected into the destination embedding space and the partial destination embedding vector is also projected into the destination embedding space. A similarity measure determines the most-similar items to the partial destination embedding vector and at least some of the most-similar items are returned in response to the input query.

Identifying similar content in a multi-item embedding space

Systems and methods for responding to a subscriber's text-based request for content items are presented. In response to a request from a subscriber, word pieces are generated from the text-based terms of the request. A request embedding vector of the word pieces is obtained from a trained machine learning model. Using the request embedding vector, a set of content items, from a corpus of content items, is identified. At least some content items of the set of content items are returned to the subscriber in response to the text-based request for content items.

Identifying content items in response to a text-based request

The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.

Efficiently augmenting images with related content

Systems and methods for recommending content to an online service subscriber are presented. For each subscriber, content items that were the subject of the subscriber's prior interactions are projected, via associated embedding vectors, into a content item embedding space. The content items, via their projections into the content item embedding space, are clustered to form a plurality of interest clusters for the subscriber. A representative embedding vector is determined for each interest cluster, and a plurality of these embedding vectors are stored as the representative embedding vectors for the subscriber. The online service, in response to a request for recommended content for a subscriber, selects a first representative embedding vector associated with the subscriber and identifies a new content item from a corpus of content items according to a similarity measure between the first representative embedding vector and an embedding vector associated with the new content item.

Recommending content to subscribers

The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A re user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.

Methods, systems, and apparatus, for determining fine-grained image similarity. In one aspect, a method includes training an image embedding function on image triplets by selecting image triplets of first, second and third images; generating, by the image embedding function, a first, second and third representations of the features of the first, second and third images; determining, based on the first representation of features and the second representation of features, a first similarity measure for the first image to the second image; determining, based on the first representation of features and the third representation of features, a second similarity measure for the the first image to the third image; determining, based on the first and second similarity measures, a performance measure of the image embedding function for the image triplet; and adjusting the parameter weights of the image embedding function based on the performance measures for the image triplets.

Fine-grained image similarity

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for analyzing image search result relevance. In one aspect, a method includes receiving result data specifying a search query and responsive image search results that reference images that are responsive to the search query. A determination is made that the search query matches an indexed query. An image relevance model is identified for the indexed query. The image relevance model can output a relevance score adjustment factor for an image search result based on image feature values of the image that is referenced by the search result. A relevance score adjustment factor is determined for each image search result using the identified image relevance model. A relevance score for each image search result is adjusted using the image's image relevance score adjustment factor. The images are ranked based on the adjusted relevance scores.

Online image analysis

A system and computer-implemented method for associating images with semantic entities and providing search results using the semantic entities. An image database contains one or more source images associated with one or more images labels. A computer may generate one or more documents containing the labels associated with each image. Analysis may be performed on the one or more documents to associate the source images with semantic entities. The semantic entities may be used to provide search results. In response to receiving a target image as a search query, the target image may be compared with the source images to identify similar images. The semantic entities associated with the similar images may be used to determine a semantic entity for the target image. The semantic entity for the target image may be used to provide search results in response to the search initiated by the target image.

System and method for associating images with semantic entities

Growing community of inventors

Cupertino, CA, United States of America

Charles Joseph Rosenberg