Microsoft Technology Licensing, LLC

Other

Microsft Technolgy Licensing, LLC

Arun Sacheti

Houdong Hu

Parthasarathy Govindarajen

Yan Wang

Li Huang

Meenaz Merchant

Yanfeng Sun

Karim Hasham

Surendra Ulabala

Kuang-Huei Lee

Aaron Chun Win Yuen

Saurajit Mukherjee

Nikita Astrakhantsev

Linjun Yang

Ye Wu

Kuansan Wang

Kun Wu

Daniel Marantz

Yu-ting Kuo

Bo Geng

William G Burton

Vishal Chhabilbhai Thakkar

Siva Mohan

Avinash Vemuluru

Justin E Cole

Ravi Theja Yada

Eugene C Chellis

Christopher VandenBerg

Pak-Ming Cheung

Fnu Yokesh Kumar

Xin Yu

Nikesh Srivastava

Henricus Johannes Maria Meijer

Heung-Yeung Shum

Xi Chelsea Chen

Josh D Benaloh

Jun Liu

Kamal Jain

Jeffrey Van Gogh

Marcelo Medeiros De Barros

Gaurang Prajapati

Danny Van Velzen

Bo-June Hsu

James Knox Russell

Xiaodong Fan

Chao-Chia Liu

Ming Ye

Diane Shambaugh

Ting Cai

Ryuichi Hirano

Deuane Martin

Achraf Chalabi

Lawrence William Colagiovanni

Malik Mehdi Pradhan

Mithun Das Gupta

Yandong Guo

Cornelia Carapcea

Nektarios Ioannides

Hua Ding

Aaron C Yuen

Andrzej Pastusiak

Dong Han Wang

Liang Tang

Soudamini Sreepada

Anjali Muralidhar

Rushabh Rajesh Gandhi

Sudhir Kumar

Darrin E Eide

Rajesh Kuppuswamy

Wallace Earl Greathouse

Yucan Zhang

Kamal Ginotra

Sumit Chatterjee

Soohoon Cho

Gautam Vishwas Vaidya

Vincent Kin-Wah Leung

Aaron Chun-Win Yuen

Guarang P Prajapati

Yin-Cheng Ting

Alexandre B R Michelis

Yogesh Ajit Vaidya

Tianjun Xiao

Nan Wu

Shloak Jain

Deepak Santhanam

Bartosz L Rakowski

Srikanth R Ravula

Yusuf Furkan Fidan

Pavel Komlev

Brad Ward

Alexandre Bernard Raymond Michelis

Alexander Jack Sunell

Alexandra Victoria Foote

Emily Lin

Elena Usinskiene

Qianyue You

Souvick Sarkar

Jiapei Huang

Adam Jeffrey Curtis

Representative embodiments disclose mechanisms to perform visual intent classification or visual intent detection or both on an image. Visual intent classification utilizes a trained machine learning model that classifies subjects in the image according to a classification taxonomy. The visual intent classification can be used as a pre-triggering mechanism to initiate further action in order to substantially save processing time. Example further actions include user scenarios, query formulation, user experience enhancement, and so forth. Visual intent detection utilizes a trained machine learning model to identify subjects in an image, place a bounding box around the image, and classify the subject according to the taxonomy. The trained machine learning model utilizes multiple feature detectors, multi-layer predictions, multilabel classifiers, and bounding box regression.

Visual intent triggering for visual search

Systems and methods can be implemented to conduct a visual search as a service in a variety of applications. In various embodiments, a system is configured to provide searching capabilities of content provided by a first entity in response to a search request by a second entity. An image provided by the second entity can be used by the system as a query image to search the content of the first entity. In an embodiment, the first entity can be a commercial entity providing such a system with image related content regarding its products and services such that any number of individual consumers can search for particular products and services of the commercial entity via their communication enabled devices. In addition, such systems can be arranged for other embodiments to provide customized searches of a single source by many individual devices. Additional systems and methods are disclosed.

Visual search services for multiple partners

A visual search system includes a computing device, where the computing device includes an image processing engine for generating a feature vector representing a user-selected object in an image. The computing device also includes, an object detection engine for locating one or more objects in the image and for determining a category of a user-selected object from objects in the image, where the object detection engine uses the category to generate a plurality of attributes for the user-selected object. The computing device further includes a product data store for storing a plurality of tables storing one or more attributes associated with a category of the user-selected object. The computing device additionally includes an attribute generation engine for generating a plurality of attribute options and an attribute matching engine for comparing attributes and attribute options of the user-selected object with attributes and attribute options of visually similar products and images.

System and method for attribute-based visual search over a computer communication network

A computing system includes a processor and memory storing instructions that, when executed by the processor, cause the processor to perform acts. The acts include receiving a query provided by a user. The acts additionally include determining that the query is related to a comparison between entities. Based upon the determining that the query is related to the comparison between entities, the computing system generates a prompt that is to be input to a generative language model, where the prompt includes:) an instruction for the generative language model to generate a table based upon the query; and) attribute values for entities identified by a search system based upon the query. The acts also include providing the prompt as input to the generative language model, where the generative language model generates a table based upon the prompt, and further where the table includes the attribute values for the entities.

Computer-implemented table construction

Systems and methods directed to returning personalized image-based search results are described. In examples, a query including an image may be received, and a personalized item embedding may be generated based on the image and user profile information associated with a user. Further, a plurality of candidate images may be obtained based on the personalized item embedding. The candidate images may then be ranked according to a predicted level of user engagement for a user, and then diversified to ensure visual diversity among the ranked images. A portion of the diversified images may then be returned in response to an image-based search.

Web-scale personalized visual search recommendation service

Described herein is a mechanism for visual recognition of items or visual search using Optical Character Recognition (OCR) of text in images. Recognized OCR blocks in an image comprise position information and recognized text. The embodiments utilize a location-aware feature vector created using the position and recognized information in each recognized block. The location-aware features of the feature vector utilize position information associated with the block to calculate a weight for the block. The recognized text is used to construct a tri-character gram frequency, inverse document frequency (TGF-IDP) metric using tri-character grams extracted from the recognized text. Features in location-aware feature vector for the block are computed by multiplying the weight and the corresponding TGF-IDF metric. The location-aware feature vector for the image is the sum of the location-aware feature vectors for the individual blocks.

Layout-aware, scalable recognition system

An image selector selects images for visual similarity in some aspects and visual distinction in other aspects. A user submits a query to a search engine. Images identified by the search engine as relevant to the query are clustered into clusters having at least one similar attribute. Selection mechanisms organize images selected from each cluster based on a similarity metric into image rows. A selection metric is calculated for each image row and a row to be displayed is selected based on the selection metric. In some embodiments, images representative of a subset of image rows are displayed to a user and responsive to selection of a representative image, the corresponding image row is displayed.

Automatic image selection for visual consistency

The present disclosure relates to systems, devices, and methods for a visual exploration of search engine results. The systems, devices, and methods may present, in response to receiving the search engine results for a query to perform a task, a search result webpage with a cascading waterfall layout that includes a plurality of images from a plurality of webpages included in the search results. The search result webpage may also present, adjacent to the plurality of images, structured data corresponding to a selected image from the plurality of images where the structured data includes details from the webpage to complete the task.

Image reranking and presentation for visual exploration

A computer-implemented technique generates a dense embedding vector that provides a distributed representation of input text. The technique includes: generating an input term-frequency (TF) vector of dimension g that includes frequency information relating to frequency of occurrence of terms in an instance of input text; using a TF-modifying component to modify the term-specific frequency information in the input TF vector by respective machine-trained weighting factors, to produce an intermediate vector of dimension g; using a projection component to project the intermediate vector of dimension g into an embedding vector of dimension k, where k is less than g. Both the TF-modifying component and the projection component use respective machine-trained neural networks. An application performs any of a retrieval-based function, a recognition-based function, a recommendation-based function, a classification-based function, etc. based on the embedding vector.

Encoder using machine-trained term frequency weighting factors that produces a dense embedding vector

The description relates to diversified hybrid image annotation for annotating images. One implementation includes generating first image annotations for a query image using a retrieval-based image annotation technique. Second image annotations can be generated for the query image using a model-based image annotation technique. The first and second image annotations can be integrated to generate a diversified hybrid image annotation result for the query image.

Image annotation

A computer-implemented technique is described herein for using a machine-trained model to identify individual objects within images. The technique then creates a relational index for the identified objects. That is, each index entry in the relational index is associated with a given object, and includes a set of attributes pertaining to the given object. One such attribute identifies at least one latent semantic vector associated with the given object. Each attribute provides a way of linking the given object to one or more other objects in the relational index. In one application of this technique, a user may submit a query that specifies a query object. The technique consults the relational index to find one or more objects that are related to the query object. In some cases, the query object and each of the other objects have a complementary relationship.

Generating and applying an object-level relational index for images

A visual search system detects one or more user-selected objects represented in an image. A first group of attributes for the user-selected objects is identified. A category for the user-selected objects is identified, and a second group of pre-defined attributes associated with the category is retrieved. The first and second groups of attributes are combined into an attributes set. The combined set of attributes are presented to the user. The user selects one or more attributes and a search is performed to identify images similar to the user-selected attributes. The images are ranked and a subset is presented to the user.

Systems and methods can be implemented to conduct searches based on images used as queries in a variety of applications. In various embodiments, a set of visual words representing a query image are generated from features extracted from the query image and are compared with visual words of index images. A set of candidate images is generated from the index images resulting from matching one or more visual words in the comparison. A multi-level ranking is conducted to sort the candidate images of the set of candidate images, and results of the multi-level ranking are returned to a user device that provided the query image. Additional systems and methods are disclosed.

Multi-modal visual search pipeline for web scale images

A visual search engine is described herein. The visual search engine is configured to return information to a client computing device based upon a multimodal query received from the client computing device (wherein the multimodal query comprises an image and text). The visual search engine is further configured to interact with a user of the client computing device to disambiguate information retrieval intent of the user.

Interactive visual search engine

Growing community of inventors

Sammamish, WA, United States of America

Arun Sacheti