Google Inc.

Ibrahim Badr

Nils Grimsmo

Gokhan Hasan Bakir

Gökhan Hasan Bakir

Zaheed Sabur

Aayush Kumar

Denis Burakov

Adrian Zumbrunnen

Roland Peter Kehl

Aneto Okonkwo

Srinivasan Venkatachary

Vladimir Vuskovic

Lucas Mirelmann

Jayant Madhavan

Matthew K Gray

John Blitzer

Yariv Adan

Mauricio Zuluaga

Mahsan Rofouei

Hugo Santos

Phiroze Parakh

Aditya Shah

Sam Oates

Corinn Herrick

Goekhan Hasan Bakir

Bianca Madalina Buisman

Behshad Behzadi

Andrea Terwisscha Van Scheltinga

Koa Metter

Fredrik Bergenlid

Quentin Lascombes De Laroussilhe

Vladyslav Lysychkin

Mikhail Golikov

Paige Alexis Dunn-Rankin

Viacheslav Kuznetsov

Kamil Anikiej

Karthik Murili Nagaraj

Glenn Wilson

Gökhan Bakir

Kavin Karthik Ilangovan

Daniel Kunkle

Michal Jastrzebski

Peter Krogh

Shikha Kapoor

Leo Deegan

Arwa Rangwala

Marcin M Nowak-Przygodzki

Ioannis Agiomyrgiannakis

Mugurel Ionut Andreica

Kaan Yücer

Michael Hatscher

Imran R Mansuri

Thomas Jonathan Stone

Paul Derek Hankin

Robin Züger

Wenqiong Guo

Piotr Malinowski

InHan Kang

Sari Onaissi

Anner Kushnir

Methods, systems, and apparatus for controlling smart devices are described. In one aspect a method includes receiving image data for an image captured by a camera of a mobile device of a user and determining that the image depicts at least one of a smart device or a physical control for the smart device. In response to determining that that the image depicts a smart device or a physical control for the smart device, identifying one or more user interface controls for controlling the smart device, and generating and presenting, at a display of the mobile device, the one or more user interface controls for controlling the smart device. The method can further include detecting, at the display of the mobile device, user interaction with at least one of the one or more user interface controls, and controlling the smart device based on the detected user interaction.

Identifying and controlling smart devices

Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.

Resolving automated assistant requests that are based on image(s) and/or other sensor data

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for providing contextual information for presented media. In one aspect, a method includes storing in a buffer, on a first user device, media data as buffered media data, the buffered media data being a most recent portion of media data received at the first user device, the most recent portion inclusive of the media data received from a present time to a prior time that is fixed relative to the present time; responsive to a search operation invocation at the present time, sending the buffered media data to a search processing system that is remote from the first user device; and receiving, from the search processing system and in response to the buffered media data, contextual information regarding an entity that the data processing system identified from processing the buffered media data.

Media contextual information for a displayed resource

Methods, apparatus, and computer readable media are described related to automated assistants that proactively incorporate, into human-to-computer dialog sessions, unsolicited content of potential interest to a user. In various implementations, in an existing human-to-computer dialog session between a user and an automated assistant, it may be determined that the automated assistant has responded to all natural language input received from the user. Based on characteristic(s) of the user, information of potential interest to the user or action(s) of potential interest to the user may be identified. Unsolicited content indicative of the information of potential interest to the user or the action(s) may be generated and incorporated by the automated assistant into the existing human-to-computer dialog session. In various implementations, the incorporating may be performed in response to the determining that the automated assistant has responded to all natural language input received from the user during the human-to-computer dialog session.

Proactive incorporation of unsolicited content into human-to-computer dialogs

Methods, apparatus, and computer readable media are described herein for allowing a first user to interface with an automated assistant to assign tasks to additional user(s), and/or for causing notification(s) of the assigned task to be rendered to the additional user(s) via corresponding automated assistant interface(s). In various implementations, one or more criteria can be utilized in selecting a group of client device(s), linked to the additional user, via which to provide the notification(s) for the task assigned to the additional user. Also, in various implementations condition(s) for providing the notification(s) for the task can be determined, and the notification(s) provided based on determining satisfaction of the condition(s).

Providing access to user-controlled resources by automated assistants

Systems and methods provide an application programming interface to offer action suggestions to third-party applications using context data associated with the third-party. An example method includes receiving content information and context information from a source mobile application, the content information representing information to be displayed on a mobile device as part of a source mobile application administered by a third party, the context information being information specific to the third party and unavailable to a screen scraper. The method also includes predicting an action based on the content information and the context information, the action representing a deep link for a target mobile application. The method further includes providing the action to the source mobile application with a title and a thumbnail, the source mobile application using the title and thumbnail to display a selectable control that, when selected, causes the mobile device to initiate the action.

Providing contextual actions for mobile onscreen content

At least selectively utilizing a large language model (LLM) in generating a natural language (NL) based summary to be rendered in response to a query. In some implementations, in generating the NL based summary additional content is processed using the LLM. The additional content is in addition to query content of the query itself and, in generating the NL based summary, can be processed using the LLM and along with the query content—or even independent of the query content. Processing the additional content can, for example, mitigate occurrences of the NL based summary including inaccuracies and/or can mitigate occurrences of the NL based summary being over-specified and/or under-specified.

Generative summaries for search results

Techniques are described herein for determining a predicted intent of a user and displaying additional content selected based on the predicted intent of the user. A method includes: receiving information identifying a webpage that a user is visiting and a navigational path of the user in navigating to the webpage; determining a predicted intent of the user based on the information identifying the webpage that the user is visiting and the navigational path of the user in navigating to the webpage; selecting additional content based upon the predicted intent of the user; and displaying an overlay, on a portion of the webpage, that includes the additional content.

Page-based prediction of user intent

A computing system and method that can be used to surface a selectable action element for at least one ongoing task. In particular, the present disclosure provides a general pipeline to identify potential tasks that a user has an ongoing interest in or has not yet completed so that a suggestion of a content item can be made to further advance an identified user's task. This pipeline can incorporate probabilistic transition graphs, machine-learned models, and/or historical data to determine the relevance and completion of tasks that a user may desire to continue acting upon.

Generating action elements suggesting content for ongoing tasks

A computer-implemented method is described. The method includes a computing system receiving an item of digital content from a user device. The computing system generates one or more labels that indicate attributes of the item of digital content. The computing system also generates one or more conversational replies to the item of digital content based on the one or more labels that indicate attributes of the item of digital content. The method also includes the computing system selecting a conversational reply from among the one or more conversational replies and providing the conversational reply for output to the user device.

Personality reply for digital content

At least selectively utilizing a large language model (LLM) in generating a natural language (NL) based summary to be rendered in response to a query. In some implementations, in generating the NL based summary additional content is processed using the LLM. The additional content is in addition to query content of the query itself and, in generating the NL based summary, can be processed using the LLM and along with the query content&#x2014;or even independent of the query content. Processing the additional content can, for example, mitigate occurrences of the NL based summary including inaccuracies and/or can mitigate occurrences of the NL based summary being over-specified and/or under-specified.

Growing community of inventors

Zurich, Switzerland

Ibrahim Badr