Google Inc.

Harshit Kharbanda

Christopher Kelley

Louis Wang

Belinda Zeng

Minsang Choi

Sabrina Curry

Jongwon Yu

Carrie Linda Bisazza

Morgane Magali Laure Sanglier

Pritam Singh Pebam

Matthew Roth

Natalie Michele Salaets

Jessica Lee

Nishant Ranka

Austin Wu

Caroline Chilton

Alok Aggarwal

Ruslan Alfridovich Abdikeev

Viviana Caso Corella

Severin Heiniger

Wenjia Yuan

Boris Bluntschli

Pendar Yousefi

Vibhuti Mahajan

Jessica Katherine Turner

Kai Yu

Dounia Berrada

Bálint Miklos

Hassan Ali Shojania

Balint Miklos

Arash Sadr

Afroz Mohiuddin

Richard Cameron

Igor Bonaci

Erica Bjornsson

Fabian Roth

Alice Au Quan

Sundeep Vaddadi

David William Hendon

Samer Hassan Hassan

Jieming Yu

Ali Essam Ali Elqursh

Daniel Valcarce Silva

Jatin Matani

Christophe Patrice Fondacci

Mikhail Khalman

Cindy L Huynh

Display screen or portion thereof with transitional graphical user interface

In a general aspect, a method can include receiving, by an electronic device, a visual scene; identifying, by the electronic device, a plurality of elements of the visual scene; and determining, based on the plurality of elements identified in the visual scene, a context of the visual scene. The method can further include applying, based on the determined context of the visual scene, at least one filter to identify at least one element of the plurality of elements corresponding with the at least one filter; and visually indicate, in the visual scene on a display of the electronic device, the at least one element identified using the at least one filter.

Finding and filtering elements of a visual scene

Systems and methods for image understanding can include one or more object recognition systems and one or more vision language models to generate an augmented language output that can be both scene-aware and object-aware. The systems and methods can process an input image with an object recognition model to generate an object recognition output descriptive of identification details for an object depicted in the input image. The systems and methods can include processing the input image with a vision language model to generate a language output descriptive of a predicted scene description. The object recognition output can then be utilized to augment the language output to generate an augmented language output that includes the scene understanding of the language output with the specificity of the object recognition output.

Instance level scene recognition with a vision language model

A multimodal search system is described. The system can receive image data captured by a camera of a user device. Additionally, the system can receive audio data associated with the image data. The audio data can be captured by a microphone of the user device. Moreover, the system can process the image data to generate visual features. Furthermore, the system can process the audio data to generate a plurality of words. The system can generate a plurality of search terms based on the plurality of words and the visual features. Subsequently, the system can determine one or more search results associated with the plurality of search terms and provide the one or more search results as an output.

Visual and audio multimodal searching system

Systems and methods for multimodal query suggestion can include obtaining image data, determining text strings associated with the image data, and providing the text strings as selectable options for performing a multimodal query with the image data. The text strings can be determined based on performing object detection and classification on the image data. The object classifications can then be leveraged for determining potential text queries a user may select for obtaining additional information about the classified object.

Multimodal query prediction

Display screen or portion thereof with graphical user interface

A multimodal search system is described. The system can receive image data from a user device. Additionally, the system can receive a prompt associated with the image data. Moreover, the system can determine, using a computer vision model, a first object in the image data that is associated with the prompt. Furthermore, the system can receive, from the user device, a user indication on whether the image data includes the first object. Subsequently, in response to receiving the user indication, the system can generate a response using a large language model.

User verification of a generative response to a multimodal query

Systems and methods for multi-image search can include obtaining two or more images and determining one or more search results that are based on the two or more images. The one or more search results can be determined based on determined shared attributes of the two or more images. The one or more search results may be based on feature embeddings associated with the two or more images. The two or more images may be obtained based on one or more user interactions with one or more databases.

Multi-image search

Systems and methods for providing visual indications of generative model responses can include obtaining a user input and processing the user input with a generative model to generate a model-generated-response. The systems and methods can process the model-generated response and an image of an environment to generate an augmented image. The augmented image can include visual indicators of the model-generated response, which can include annotating the image based on detected features within the image. Generation of the augmented image can include object detection and annotation based on the content of the model-generated response.

Visual indicators of generative model response details

Systems and methods for providing scene understanding can include obtaining a plurality of images, stitching images associated with the scene, detecting objects in the scene, and providing information associated with the objects in the scene. The systems and methods can include determining filter tags or query tags that can be selected to filter the plurality of objects, which can then be provided as information to the user to provide further insight on the scene. The information may be provided in an augmented-reality experience via text or other user-interface elements anchored to objects in the images.

Object filtering and information display in an augmented-reality experience

Systems and methods for textual replacement can include the determination of a visual intent, which can trigger an interface for selecting an image to replace visual descriptors. The visually descriptive terms can be identified, and an indicator can be provided to indicate the text replacement option may be initiated. An image can then be selected by a user to replace the visually descriptive terms.

Visual search determination for text-to-image replacement

Growing community of inventors

Pleasanton, CA, United States of America

Harshit Kharbanda