Google Inc.

Planet Labs, Inc.

Christopher Kelley

Harshit Kharbanda

Louis Wang

Belinda Zeng

Minsang Choi

Sabrina Curry

Matthew Roth

Natalie Michele Salaets

Pritam Singh Pebam

Jongwon Yu

Morgane Magali Laure Sanglier

Carrie Linda Bisazza

Nishant Ranka

Austin Wu

Caroline Chilton

Jessica Lee

Alok Chandel

George Cody Sumter

Matthew David Tait

Shane Riley Brennan

Pendar Yousefi

Dounia Berrada

Alok Aggarwal

Balint Miklos

Julian M Mann

Daniel Berkenstock

Adam Thielemann Wegel

Oliver Guinan

Arash Sadr

Viviana Caso Corella

Severin Heiniger

Igor Bonaci

Erica Bjornsson

Alice Au Quan

Afroz Mohiuddin

Fabian Roth

James Long

Sundeep Vaddadi

David William Hendon

Kai Yu

Samer Hassan Hassan

Richard J Cirminello, Jr

Mikhail Khalman

Ching-Yu Hu

Daniel Valcarce Silva

Jieming Yu

Greg Bertoldo

Ali Essam Ali Elqursh

Carlos Cervantes

In a general aspect, a method can include receiving, by an electronic device, a visual scene; identifying, by the electronic device, a plurality of elements of the visual scene; and determining, based on the plurality of elements identified in the visual scene, a context of the visual scene. The method can further include applying, based on the determined context of the visual scene, at least one filter to identify at least one element of the plurality of elements corresponding with the at least one filter; and visually indicate, in the visual scene on a display of the electronic device, the at least one element identified using the at least one filter.

Finding and filtering elements of a visual scene

A multimodal search system is described. The system can receive image data captured by a camera of a user device. Additionally, the system can receive audio data associated with the image data. The audio data can be captured by a microphone of the user device. Moreover, the system can process the image data to generate visual features. Furthermore, the system can process the audio data to generate a plurality of words. The system can generate a plurality of search terms based on the plurality of words and the visual features. Subsequently, the system can determine one or more search results associated with the plurality of search terms and provide the one or more search results as an output.

Visual and audio multimodal searching system

Display screen or portion thereof with graphical user interface

Display screen or portion thereof with transitional graphical user interface

A multimodal search system is described. The system can receive image data from a user device. Additionally, the system can receive a prompt associated with the image data. Moreover, the system can determine, using a computer vision model, a first object in the image data that is associated with the prompt. Furthermore, the system can receive, from the user device, a user indication on whether the image data includes the first object. Subsequently, in response to receiving the user indication, the system can generate a response using a large language model.

User verification of a generative response to a multimodal query

Systems and methods for multi-image search can include obtaining two or more images and determining one or more search results that are based on the two or more images. The one or more search results can be determined based on determined shared attributes of the two or more images. The one or more search results may be based on feature embeddings associated with the two or more images. The two or more images may be obtained based on one or more user interactions with one or more databases.

Multi-image search

Systems and methods for providing visual indications of generative model responses can include obtaining a user input and processing the user input with a generative model to generate a model-generated-response. The systems and methods can process the model-generated response and an image of an environment to generate an augmented image. The augmented image can include visual indicators of the model-generated response, which can include annotating the image based on detected features within the image. Generation of the augmented image can include object detection and annotation based on the content of the model-generated response.

Visual indicators of generative model response details

Systems and methods for providing scene understanding can include obtaining a plurality of images, stitching images associated with the scene, detecting objects in the scene, and providing information associated with the objects in the scene. The systems and methods can include determining filter tags or query tags that can be selected to filter the plurality of objects, which can then be provided as information to the user to provide further insight on the scene. The information may be provided in an augmented-reality experience via text or other user-interface elements anchored to objects in the images.

Object filtering and information display in an augmented-reality experience

Systems and methods for textual replacement can include the determination of a visual intent, which can trigger an interface for selecting an image to replace visual descriptors. The visually descriptive terms can be identified, and an indicator can be provided to indicate the text replacement option may be initiated. An image can then be selected by a user to replace the visually descriptive terms.

Visual search determination for text-to-image replacement

Growing community of inventors

Orinda, CA, United States of America

Christopher Kelley