Microsoft Technology Licensing, LLC

Li Huang

Houdong Hu

Arun Sacheti

Meenaz Merchant

Congyong Su

Yan Wang

Linjun Yang

Kuang-Huei Lee

Saurajit Mukherjee

Huangxing Li

Xi Chelsea Chen

Ye Wu

Kamal Ginotra

Lu Li

Xi Chen

Sean Chang Culatana

Rui Xia

Jiapei Huang

Hong Xuan

Hanmu Zuo

Andrew Lawrence Stewart

Kun Wu

Chu Wang

Zhiting Chen

Tianjun Xiao

Yokesh Kumar

Aspects of the disclosure include methods and systems for leveraging a controllable diffusion model for dynamic image search in an image gallery recommendation service. An exemplary method can include displaying an image gallery having a plurality of gallery images and a dynamic image frame. The dynamic image frame can include a generated image and an interactive widget. The method can include receiving a user input in the interactive widget and generating, responsive to receiving the user input, an updated generated image by inputting, into a controllable diffusion model, the user input. The method can include replacing the generated image in the dynamic image frame with the updated generated image.

Controllable diffusion model based image gallery recommendation service

Representative embodiments disclose mechanisms to perform visual intent classification or visual intent detection or both on an image. Visual intent classification utilizes a trained machine learning model that classifies subjects in the image according to a classification taxonomy. The visual intent classification can be used as a pre-triggering mechanism to initiate further action in order to substantially save processing time. Example further actions include user scenarios, query formulation, user experience enhancement, and so forth. Visual intent detection utilizes a trained machine learning model to identify subjects in an image, place a bounding box around the image, and classify the subject according to the taxonomy. The trained machine learning model utilizes multiple feature detectors, multi-layer predictions, multilabel classifiers, and bounding box regression.

Visual intent triggering for visual search

Aspects of the disclosure include methods and systems for dynamically creating tagged search results. An exemplary method includes identifying, responsive to receiving a search query, search results. The search results include a plurality of web page links to respective web pages. The method includes identifying, for each web page, one or more page features, generating a plurality of descriptive tags for the respective web pages, generating one or more page-tag pairs by pairing a web page of the respective web pages with a descriptive tag of the plurality of descriptive tags, and determining a relevance score for each page-tag pair. One or more descriptive tags having a relevance score above a predetermined threshold are selected. Tagged search results are provided that include the plurality of web page links and, for each web page link, the selected one or more descriptive tags for the respective web page.

Dynamic visual tagging system for search

Systems and methods can be implemented to conduct a visual search as a service in a variety of applications. In various embodiments, a system is configured to provide searching capabilities of content provided by a first entity in response to a search request by a second entity. An image provided by the second entity can be used by the system as a query image to search the content of the first entity. In an embodiment, the first entity can be a commercial entity providing such a system with image related content regarding its products and services such that any number of individual consumers can search for particular products and services of the commercial entity via their communication enabled devices. In addition, such systems can be arranged for other embodiments to provide customized searches of a single source by many individual devices. Additional systems and methods are disclosed.

Visual search services for multiple partners

A visual search system includes a computing device, where the computing device includes an image processing engine for generating a feature vector representing a user-selected object in an image. The computing device also includes, an object detection engine for locating one or more objects in the image and for determining a category of a user-selected object from objects in the image, where the object detection engine uses the category to generate a plurality of attributes for the user-selected object. The computing device further includes a product data store for storing a plurality of tables storing one or more attributes associated with a category of the user-selected object. The computing device additionally includes an attribute generation engine for generating a plurality of attribute options and an attribute matching engine for comparing attributes and attribute options of the user-selected object with attributes and attribute options of visually similar products and images.

System and method for attribute-based visual search over a computer communication network

A computer-implemented technique uses a machine-trained model to convert input information provided by a user into a natural language expression and a semantic vector. The semantic vector provides a distributed representation of the natural language expression. The technique then uses the generated semantic vector to find a target data record having a target semantic vector that is within a prescribed distance to the generated semantic vector. The technique then outputs at least a target natural language expression that is associated with the target data record. The target data record is produced in an offline process that converts one or more attribute values in an existing structured data record into the target semantic vector and the target natural language expression. Also described herein is a technique for training the machine-trained model and for updating existing collections of data records using the machine-trained model.

Creating and interacting with data records having semantic vectors and natural language expressions produced by a machine-trained model

Systems and methods directed to returning personalized image-based search results are described. In examples, a query including an image may be received, and a personalized item embedding may be generated based on the image and user profile information associated with a user. Further, a plurality of candidate images may be obtained based on the personalized item embedding. The candidate images may then be ranked according to a predicted level of user engagement for a user, and then diversified to ensure visual diversity among the ranked images. A portion of the diversified images may then be returned in response to an image-based search.

Web-scale personalized visual search recommendation service

The description relates to diversified hybrid image annotation for annotating images. One implementation includes generating first image annotations for a query image using a retrieval-based image annotation technique. Second image annotations can be generated for the query image using a model-based image annotation technique. The first and second image annotations can be integrated to generate a diversified hybrid image annotation result for the query image.

Image annotation

A visual search system detects one or more user-selected objects represented in an image. A first group of attributes for the user-selected objects is identified. A category for the user-selected objects is identified, and a second group of pre-defined attributes associated with the category is retrieved. The first and second groups of attributes are combined into an attributes set. The combined set of attributes are presented to the user. The user selects one or more attributes and a search is performed to identify images similar to the user-selected attributes. The images are ranked and a subset is presented to the user.

Systems and methods can be implemented to conduct searches based on images used as queries in a variety of applications. In various embodiments, a set of visual words representing a query image are generated from features extracted from the query image and are compared with visual words of index images. A set of candidate images is generated from the index images resulting from matching one or more visual words in the comparison. A multi-level ranking is conducted to sort the candidate images of the set of candidate images, and results of the multi-level ranking are returned to a user device that provided the query image. Additional systems and methods are disclosed.

Multi-modal visual search pipeline for web scale images

A visual search engine is described herein. The visual search engine is configured to return information to a client computing device based upon a multimodal query received from the client computing device (wherein the multimodal query comprises an image and text). The visual search engine is further configured to interact with a user of the client computing device to disambiguate information retrieval intent of the user.

Interactive visual search engine

Methods, systems, and computer programs are presented for identifying the brand and model of products embedded within an image. One method includes operations for receiving, via a graphical user interface (GUI), a selection of an image, and for analyzing the image to determine a location within the image of one or more products. For each product in the image, determining a unique identification of the product is determined, the unique identification including a manufacturer of the product and a model identifier. The method further includes an operation for presenting information about the one or more products in the GUI with a selection option for selecting each of the one or more products. Additionally, the method includes operations for receiving a product selection for one of the one or more products, and presenting shopping options in the GUI for purchasing the selected product.

Product identification in image with multiple products

A technique is described herein for transforming audio content into images. The technique may include: receiving the audio content from a source; converting the audio content into a temporal stream of audio features; and converting the stream of audio features into one or more images using one or more machine-trained models. The technique generates the image(s) based on recognition of: semantic information that conveys one or more semantic topics associated with the audio content; and sentiment information that conveys one or more sentiments associated with the audio content. The technique then generates an output presentation that includes the image(s), which it provides to one or more display devices for display thereat. The output presentation serves as a summary of salient semantic and sentiment-related characteristics of the audio content.

Growing community of inventors

Sammamish, WA, United States of America

Li Huang