Microsoft Technology Licensing, LLC

Houdong Hu

Arun Sacheti

Li Huang

Meenaz Merchant

Yan Wang

Kuang-Huei Lee

Xi Chelsea Chen

Linjun Yang

Ye Wu

Julia Gong

Gang Hua

Kun Wu

William Douglas Guyman

Surendra Ulabala

Congyong Su

Mithun Das Gupta

Saurajit Mukherjee

Vishal Chhabilbhai Thakkar

Jyotkumar Jagdishbhai Patel

Xuetao Yin

Rushabh Rajesh Gandhi

Sudhir Kumar

Alexey Volkov

Soudamini Sreepada

Yale Song

Xiujia Guo

Rajiv S Binwade

Jiapei Huang

Tianjun Xiao

Rui Xia

Sean Chang Culatana

He Xiaodong

Xi Chen

Mark Robert Bolin

Ning Ma

Aleksandr Livshits

Yokesh Kumar

Yiran Shen

Pawel Michal Pietrusinksi

Representative embodiments disclose mechanisms to perform visual intent classification or visual intent detection or both on an image. Visual intent classification utilizes a trained machine learning model that classifies subjects in the image according to a classification taxonomy. The visual intent classification can be used as a pre-triggering mechanism to initiate further action in order to substantially save processing time. Example further actions include user scenarios, query formulation, user experience enhancement, and so forth. Visual intent detection utilizes a trained machine learning model to identify subjects in an image, place a bounding box around the image, and classify the subject according to the taxonomy. The trained machine learning model utilizes multiple feature detectors, multi-layer predictions, multilabel classifiers, and bounding box regression.

Visual intent triggering for visual search

Systems and methods can be implemented to conduct a visual search as a service in a variety of applications. In various embodiments, a system is configured to provide searching capabilities of content provided by a first entity in response to a search request by a second entity. An image provided by the second entity can be used by the system as a query image to search the content of the first entity. In an embodiment, the first entity can be a commercial entity providing such a system with image related content regarding its products and services such that any number of individual consumers can search for particular products and services of the commercial entity via their communication enabled devices. In addition, such systems can be arranged for other embodiments to provide customized searches of a single source by many individual devices. Additional systems and methods are disclosed.

Visual search services for multiple partners

A data processing system implements a receiving a textual input comprising a query for a first image. The data processing system also implements analyzing the textual input to determine a predicted color palette associated with a subject matter of the query; and procedurally generating the first image using the predicted color palette. Another implementation of the data processing system implements providing the textual input to a first machine learning model to obtain the first image, the first machine learning model being trained using a dataset comprising abstract imagery and analyzing the textual input using the first machine learning model to obtain the first image in response to receiving the textual input.

Techniques for abstract image generation from multimodal inputs with content appropriateness considerations

A visual search system includes a computing device, where the computing device includes an image processing engine for generating a feature vector representing a user-selected object in an image. The computing device also includes, an object detection engine for locating one or more objects in the image and for determining a category of a user-selected object from objects in the image, where the object detection engine uses the category to generate a plurality of attributes for the user-selected object. The computing device further includes a product data store for storing a plurality of tables storing one or more attributes associated with a category of the user-selected object. The computing device additionally includes an attribute generation engine for generating a plurality of attribute options and an attribute matching engine for comparing attributes and attribute options of the user-selected object with attributes and attribute options of visually similar products and images.

System and method for attribute-based visual search over a computer communication network

The present disclosure provides method and apparatus for determining a food item from a photograph and a corresponding restaurant serving the food item. An image is received from a user, the image being associated with a consumable item. One or more ingredients of the consumable item in the image is identified along with a location of the user and using a neural network, determining one or more similar images from a database. A restaurant associated with each of the one or more similar images is determined along with a similarity score indicating a similarity between the restaurant and the identified content of the image. The one or more restaurants and/or associated similar food items are ranked based on the similarity score and a list of ranked restaurants is provided to the user.

Providing local recommendations based on images of consumable items

A computer-implemented technique generates a dense embedding vector that provides a distributed representation of input text. The technique includes: generating an input term-frequency (TF) vector of dimension g that includes frequency information relating to frequency of occurrence of terms in an instance of input text; using a TF-modifying component to modify the term-specific frequency information in the input TF vector by respective machine-trained weighting factors, to produce an intermediate vector of dimension g; using a projection component to project the intermediate vector of dimension g into an embedding vector of dimension k, where k is less than g. Both the TF-modifying component and the projection component use respective machine-trained neural networks. An application performs any of a retrieval-based function, a recognition-based function, a recommendation-based function, a classification-based function, etc. based on the embedding vector.

Encoder using machine-trained term frequency weighting factors that produces a dense embedding vector

The description relates to diversified hybrid image annotation for annotating images. One implementation includes generating first image annotations for a query image using a retrieval-based image annotation technique. Second image annotations can be generated for the query image using a model-based image annotation technique. The first and second image annotations can be integrated to generate a diversified hybrid image annotation result for the query image.

Image annotation

A computer-implemented technique is described herein for using a machine-trained model to identify individual objects within images. The technique then creates a relational index for the identified objects. That is, each index entry in the relational index is associated with a given object, and includes a set of attributes pertaining to the given object. One such attribute identifies at least one latent semantic vector associated with the given object. Each attribute provides a way of linking the given object to one or more other objects in the relational index. In one application of this technique, a user may submit a query that specifies a query object. The technique consults the relational index to find one or more objects that are related to the query object. In some cases, the query object and each of the other objects have a complementary relationship.

Generating and applying an object-level relational index for images

A visual search system detects one or more user-selected objects represented in an image. A first group of attributes for the user-selected objects is identified. A category for the user-selected objects is identified, and a second group of pre-defined attributes associated with the category is retrieved. The first and second groups of attributes are combined into an attributes set. The combined set of attributes are presented to the user. The user selects one or more attributes and a search is performed to identify images similar to the user-selected attributes. The images are ranked and a subset is presented to the user.

The present concepts relate to matching data of two different modalities using two stages of attention. First data is encoded as a set of first vectors representing components of the first data, and second data is encoded as a set of second vectors representing components of the second data. In the first stage, the components of the first data are attended by comparing the first vectors and the second vectors to generate a set of attended vectors. In the second stage, the components of the second data are attended by comparing the second vectors and the attended vectors to generate a plurality of relevance scores. Then, the relevance scores are pooled to calculate a similarity score that indicates a degree of similarity between the first data and the second data.

Stacked cross-modal matching

Systems and methods can be implemented to conduct searches based on images used as queries in a variety of applications. In various embodiments, a set of visual words representing a query image are generated from features extracted from the query image and are compared with visual words of index images. A set of candidate images is generated from the index images resulting from matching one or more visual words in the comparison. A multi-level ranking is conducted to sort the candidate images of the set of candidate images, and results of the multi-level ranking are returned to a user device that provided the query image. Additional systems and methods are disclosed.

Multi-modal visual search pipeline for web scale images

A visual search engine is described herein. The visual search engine is configured to return information to a client computing device based upon a multimodal query received from the client computing device (wherein the multimodal query comprises an image and text). The visual search engine is further configured to interact with a user of the client computing device to disambiguate information retrieval intent of the user.

Interactive visual search engine

Methods, systems, and computer programs are presented for identifying the brand and model of products embedded within an image. One method includes operations for receiving, via a graphical user interface (GUI), a selection of an image, and for analyzing the image to determine a location within the image of one or more products. For each product in the image, determining a unique identification of the product is determined, the unique identification including a manufacturer of the product and a model identifier. The method further includes an operation for presenting information about the one or more products in the GUI with a selection option for selecting each of the one or more products. Additionally, the method includes operations for receiving a product selection for one of the one or more products, and presenting shopping options in the GUI for purchasing the selected product.

Product identification in image with multiple products

Systems and methods for identifying search results in response to a search query are presented. More particularly, images are selected as search results, at least in part, according to an attractiveness value associated with the images. Upon receiving a search query, a set of content is identified according to the query intent of the search query and includes at least one image. The identified set of content is ordered according an overall score determined according to relevance and, in the case of the at least one image, according to an attractiveness value. A search results generator selects items from the set of content according to their overall scores, including the at least one image, generates a search results page, and returns the search results page to the requesting party.

Search results through image attractiveness

A technique is described herein for transforming audio content into images. The technique may include: receiving the audio content from a source; converting the audio content into a temporal stream of audio features; and converting the stream of audio features into one or more images using one or more machine-trained models. The technique generates the image(s) based on recognition of: semantic information that conveys one or more semantic topics associated with the audio content; and sentiment information that conveys one or more sentiments associated with the audio content. The technique then generates an output presentation that includes the image(s), which it provides to one or more display devices for display thereat. The output presentation serves as a summary of salient semantic and sentiment-related characteristics of the audio content.

Growing community of inventors

Redmond, WA, United States of America

Houdong Hu