Google Inc.

Gökhan Bakir

Marcin M Nowak-Przygodzki

Ibrahim Badr

Gabriel Taubman

Ilya Firman

Adam Coimbra

Nils Grimsmo

James Stout

Joseph Lange

Thomas Weedon Hume

Abhanshu Sharma

Aneto Okonkwo

Matthew Stokes

Jindong Chen

Mauricio Zuluaga

Mugurel-Ionut Andreica

Andre Elisseeff

Dana Ritter

Jacek Szmigiel

Kyle Pedersen

Torsten Marek

João Paulo Pagaime Da Silva

Mathias Carlen

Lukasz Suder

Ernest Galbrun

Reed Enger

Vishwath Mohan

Yunfan Jin

Gilles Baechler

Marius Dumitran

Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.

Resolving automated assistant requests that are based on image(s) and/or other sensor data

Generating and/or utilizing image shortcuts that cause one or more corresponding computer actions to be performed in response to determining that one or more features are present in image(s) from a camera of a computing device of a user (e.g., present in a real-time image feed from the camera). An image shortcut can be generated in response to user interface input, such as a spoken command. For example, the user interface input can direct the automated assistant to perform one or more actions in response to object(s) having certain feature(s) being present in a field of view of the camera. Subsequently, when the user directs their camera at object(s) having such feature(s), the assistant application can cause the action(s) to be automatically performed. For example, the assistant application can cause data to be presented and/or can control a remote device in accordance with the image shortcut.

Systems, methods, and apparatus for providing image shortcuts for an assistant application

Techniques described herein enable a user to interact with an automated assistant and obtain relevant output from the automated assistant without requiring arduous typed input to be provided by the user and/or without requiring the user to provide spoken input that could cause privacy concerns (e.g., if other individuals are nearby). The assistant application can operate in multiple different image conversation modes in which the assistant application is responsive to various objects in a field of view of the camera. The image conversation modes can be suggested to the user when a particular object is detected in the field of view of the camera. When the user selects an image conversation mode, the assistant application can thereafter provide output, for presentation, that is based on the selected image conversation mode and that is based on object(s) captured by image(s) of the camera.

Systems, methods, and apparatus for image-responsive automated assistants

A method includes receiving, via a user interface of a client device, a request to populate one or more cells of a plurality of cells of a document having a tabular structure, wherein the one or more cells correspond to a first attribute pertaining to a first column header and a first object pertaining to a first row header; analyzing the request and one or more additional cells corresponding to one or more additional attributes and one or more additional objects of the document to obtain contextual information for the request; generating a query based at least in part on the contextual information; initiating an execution of the query to obtain a response using one or more data sources; and causing the user interface to be modified to populate the response in the one or more cells corresponding to the first attribute and the first object.

Contextual answer generation in spreadsheets

Systems and methods provide an application programming interface to offer action suggestions to third-party applications using context data associated with the third-party. An example method includes receiving content information and context information from a source mobile application, the content information representing information to be displayed on a mobile device as part of a source mobile application administered by a third party, the context information being information specific to the third party and unavailable to a screen scraper. The method also includes predicting an action based on the content information and the context information, the action representing a deep link for a target mobile application. The method further includes providing the action to the source mobile application with a title and a thumbnail, the source mobile application using the title and thumbnail to display a selectable control that, when selected, causes the mobile device to initiate the action.

Providing contextual actions for mobile onscreen content

Implementations are described herein for analyzing existing interactive web sites to facilitate automatic engagement with those web sites, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those websites. For example, in various implementations, techniques described herein may be used to extract, validate, maintain, generalize, extend and/or distribute individual actions and 'traces' of actions that are useable to navigate through various interactive websites. Additionally, techniques are described herein for leveraging these actions and/or traces to automate aspects of interaction with a third party website. For example, in some implementations, techniques described herein may enable users to engage with an automated assistant (via a spoken or typed dialog session) to interact with the third party website without requiring the user to visually interact with the third party web site directly and without requiring the third party to implement their own third party agent.

Analyzing web pages to facilitate automatic navigation

Implementations set forth herein relate to an automated assistant that can interact with applications that may not have been pre-configured for interfacing with the automated assistant. The automated assistant can identify content of an application interface of the application to determine synonymous terms that a user may speak when commanding the automated assistant to perform certain tasks. Speech processing operations employed by the automated assistant can be biased towards these synonymous terms when the user is accessing an application interface of the application. In some implementations, the synonymous terms can be identified in a responsive language of the automated assistant when the content of the application interface is being rendered in a different language. This can allow the automated assistant to operate as an interface between the user and certain applications that may not be rendering content in a native language of the user.

Automated assistant control of non-assistant applications via identification of synonymous term and/or speech processing biasing

Growing community of inventors

Zurich, Switzerland

Gökhan Bakir