Microsoft Technology Licensing, LLC

Owais Khan Mohammed

Bradley Moore Abrams

Kamal Ginotra

Weiming Wen

Michael Lawrence Valenzuela

Barun Patra

Shubham Agrawal

Vishrav Chaudhary

The disclosure describes utilizing an image query system to improve response accuracy and reduce computational steps resources in responding to natural language queries of input images. In various implementations, the image query system utilizes grounding information from one or more sources to determine accurate information for an input image. For example, the image query system uses a single comprehensive image prompt to obtain extensive visual image grounding information for the input image from a visual-based large generative model. Additionally, or in alternative implementations, the image query system obtains reverse image search grounding information for the input image. The image query system then cleverly utilizes the grounding information with a large generative language model to generate text query responses to image-based queries of the input image more accurately and efficiently.

Using large generative models with improved grounding to improve image context queries

Technology is described herein for generating questions using a neural network. The technology generates the questions in a three-step process. In the first step, the technology selects, using a first neural network, a subset of textual passages from an identified electronic document. In the second step, the technology generates, using a second neural network, one or more candidate answers for each textual passage selected by the first neural network, to produce a plurality of candidate passage-answer pairs. In the third step, the technology selects, using a third neural network, a subset of the plurality of candidate passage-answer pairs. The technology then generates an output result that includes one or more output questions chosen from the candidate passage-answer pairs produced by the third neural network. The use of the first neural network reduces the processing burden placed on the second and third neural networks. It also reduces latency.

Generating questions using a resource-efficient neural network

Growing community of inventors

Bellevue, WA, United States of America

Owais Khan Mohammed