Microsoft Technology Licensing, LLC

Kamal Ginotra

Arun Sacheti

Li Huang

Bradley Moore Abrams

Meenaz Merchant

Saurajit Mukherjee

Owais Khan Mohammed

Rui Xia

Chu Wang

Kun Wu

Andrew Lawrence Stewart

Hanmu Zuo

Michael Lawrence Valenzuela

Barun Patra

Zhiting Chen

Vishrav Chaudhary

The disclosure describes utilizing an image query system to improve response accuracy and reduce computational steps resources in responding to natural language queries of input images. In various implementations, the image query system utilizes grounding information from one or more sources to determine accurate information for an input image. For example, the image query system uses a single comprehensive image prompt to obtain extensive visual image grounding information for the input image from a visual-based large generative model. Additionally, or in alternative implementations, the image query system obtains reverse image search grounding information for the input image. The image query system then cleverly utilizes the grounding information with a large generative language model to generate text query responses to image-based queries of the input image more accurately and efficiently.

Using large generative models with improved grounding to improve image context queries

Systems and methods directed to returning personalized image-based search results are described. In examples, a query including an image may be received, and a personalized item embedding may be generated based on the image and user profile information associated with a user. Further, a plurality of candidate images may be obtained based on the personalized item embedding. The candidate images may then be ranked according to a predicted level of user engagement for a user, and then diversified to ensure visual diversity among the ranked images. A portion of the diversified images may then be returned in response to an image-based search.

Web-scale personalized visual search recommendation service

Growing community of inventors

Kirkland, WA, United States of America

Kamal Ginotra