Google Inc.

Louis Wang

Harshit Kharbanda

Christopher Kelley

Belinda Zeng

Sabrina Curry

Minsang Choi

Jongwon Yu

Natalie Michele Salaets

Matthew Roth

Carrie Linda Bisazza

Pritam Singh Pebam

Morgane Magali Laure Sanglier

Nishant Ranka

Austin Wu

Caroline Chilton

Jessica Lee

Charles Joseph Rosenberg

Charles Yang

Boris Bluntschli

Kai Yu

Vibhuti Mahajan

Balint Miklos

Wenjia Yuan

Pendar Yousefi

Ruslan Alfridovich Abdikeev

Igor Bonaci

Richard Cameron

Viviana Caso Corella

Severin Heiniger

Sundeep Vaddadi

Dounia Berrada

Christophe Patrice Fondacci

Cindy L Huynh

Daniel Valcarce Silva

Jatin Matani

In a general aspect, a method can include receiving, by an electronic device, a visual scene; identifying, by the electronic device, a plurality of elements of the visual scene; and determining, based on the plurality of elements identified in the visual scene, a context of the visual scene. The method can further include applying, based on the determined context of the visual scene, at least one filter to identify at least one element of the plurality of elements corresponding with the at least one filter; and visually indicate, in the visual scene on a display of the electronic device, the at least one element identified using the at least one filter.

Finding and filtering elements of a visual scene

Systems and methods for image understanding can include one or more object recognition systems and one or more vision language models to generate an augmented language output that can be both scene-aware and object-aware. The systems and methods can process an input image with an object recognition model to generate an object recognition output descriptive of identification details for an object depicted in the input image. The systems and methods can include processing the input image with a vision language model to generate a language output descriptive of a predicted scene description. The object recognition output can then be utilized to augment the language output to generate an augmented language output that includes the scene understanding of the language output with the specificity of the object recognition output.

Instance level scene recognition with a vision language model

A multimodal search system is described. The system can receive image data captured by a camera of a user device. Additionally, the system can receive audio data associated with the image data. The audio data can be captured by a microphone of the user device. Moreover, the system can process the image data to generate visual features. Furthermore, the system can process the audio data to generate a plurality of words. The system can generate a plurality of search terms based on the plurality of words and the visual features. Subsequently, the system can determine one or more search results associated with the plurality of search terms and provide the one or more search results as an output.

Visual and audio multimodal searching system

Systems and methods for multimodal query suggestion can include obtaining image data, determining text strings associated with the image data, and providing the text strings as selectable options for performing a multimodal query with the image data. The text strings can be determined based on performing object detection and classification on the image data. The object classifications can then be leveraged for determining potential text queries a user may select for obtaining additional information about the classified object.

Multimodal query prediction

Display screen or portion thereof with graphical user interface

Display screen or portion thereof with transitional graphical user interface

A multimodal search system is described. The system can receive image data from a user device. Additionally, the system can receive a prompt associated with the image data. Moreover, the system can determine, using a computer vision model, a first object in the image data that is associated with the prompt. Furthermore, the system can receive, from the user device, a user indication on whether the image data includes the first object. Subsequently, in response to receiving the user indication, the system can generate a response using a large language model.

User verification of a generative response to a multimodal query

Systems and methods for providing visual indications of generative model responses can include obtaining a user input and processing the user input with a generative model to generate a model-generated-response. The systems and methods can process the model-generated response and an image of an environment to generate an augmented image. The augmented image can include visual indicators of the model-generated response, which can include annotating the image based on detected features within the image. Generation of the augmented image can include object detection and annotation based on the content of the model-generated response.

Visual indicators of generative model response details

Growing community of inventors

San Francisco, CA, United States of America

Louis Wang