Google Inc.

Gökhan Hasan Bakir

Marcin M Nowak-Przygodzki

Ibrahim Badr

Aneto Okonkwo

Behshad Behzadi

Nils Grimsmo

Mauricio Zuluaga

Károly Csalogány

Denis Burakov

Daniel Kunkle

Kavin Karthik Ilangovan

Michael William Springer

Fan Jiang

Ashutosh Kulshreshtha

Vladimir Vuskovic

Mugurel Ionut Andreica

Matthew Stokes

Patrick Pui Wah Leung

Mugurel-Ionut Andreica

Andre Elisseeff

Dana Ritter

Malgorzata Monika Lisowiec

Torsten Marek

João Paulo Pagaime Da Silva

Mathias Carlen

Lukasz Suder

Ernest Galbrun

Marius Dumitran

Methods, apparatus, systems, and computer-readable media are set forth for generating and/or utilizing image shortcuts that cause one or more corresponding computer actions to be performed in response to determining that one or more features are present in image(s) from a camera of a computing device of a user (e.g., present in a real-time image feed from the camera). An image shortcut can be generated in response to user interface input, such as a spoken command. For example, the user interface input can direct the automated assistant to perform one or more actions in response to object(s) having certain feature(s) being present in a field of view of the camera. Subsequently, when the user directs their camera at object(s) having such feature(s), the assistant application can cause the action(s) to be automatically performed. For example, the assistant application can cause data to be presented and/or can control a remote device in accordance with the image shortcut.

Systems, methods, and apparatus for providing image shortcuts for an assistant application

Techniques for contextual search on multimedia content are provided. An example method includes extracting entities associated with multimedia content, wherein the entities include values characterizing one or more objects represented in the multimedia content, generating one or more query rewrite candidates based on the extracted entities and one or more terms in a query related to the multimedia content, providing the one or more query rewrite candidates to a search engine, scoring the one or more query rewrite candidates, ranking the scored one or more query rewrite candidates based on their respective scores, rewriting the query related to the multimedia content based on a particular ranked query rewrite candidate and providing for display, responsive to the query related to the multimedia content, a result set from the search engine based on the rewritten query.

Contextual search on multimedia content

Systems and methods provide an application programming interface to offer action suggestions to third-party applications using context data associated with the third-party. An example method includes receiving content information and context information from a source mobile application, the content information representing information to be displayed on a mobile device as part of a source mobile application administered by a third party, the context information being information specific to the third party and unavailable to a screen scraper. The method also includes predicting an action based on the content information and the context information, the action representing a deep link for a target mobile application. The method further includes providing the action to the source mobile application with a title and a thumbnail, the source mobile application using the title and thumbnail to display a selectable control that, when selected, causes the mobile device to initiate the action.

Providing contextual actions for mobile onscreen content

Methods, apparatus, and computer readable media are described related to causing processing of sensor data to be performed in response to determining a request related to an environmental object that is likely captured by the sensor data. Some implementations further relate to determining whether the request is resolvable based on the processing of the sensor data. When it is determined that the request is not resolvable, a prompt is determined and provided as user interface output, where the prompt provides guidance on further input that will enable the request to be resolved. In those implementations, the further input (e.g., additional sensor data and/or the user interface input) received in response to the prompt can then be utilized to resolve the request.

Resolving automated assistant requests that are based on image(s) and/or other sensor data

A method includes receiving, via a user interface of a device associated with a user, a request to populate one or more cells of a plurality of cells of a document having a tabular structure, wherein the one or more cells correspond to a first attribute pertaining to a first column and a first object pertaining to a first row; analyzing the request to obtain contextual information indicating the first attribute and the first object; generating a query based at least in part on the contextual information; initiating an execution of the query to obtain a response using one or more data sources; causing the user interface to be modified to populate the response in the one or more cells corresponding to the first attribute and the first object; determining second contextual information based on the response, the second contextual information indicating a second attribute and a second object; generating a second query based at least in part on the second contextual information; initiating an execution of the second query to obtain a second response using one or more data sources; and causing the user interface to be modified to populate the second response in one or more cells corresponding to the second attribute and the second object.

Contextual answer generation in spreadsheets

Implementations are described herein for analyzing existing interactive web sites to facilitate automatic engagement with those web sites, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those websites. For example, in various implementations, techniques described herein may be used to abstract, validate, maintain, generalize, extend and/or distribute individual actions and 'traces' of actions that are useable to navigate through various interactive websites. Additionally, techniques are described herein for leveraging these actions and/or traces to automate aspects of interaction with a third party website. For example, in some implementations, techniques described herein may enable users to engage with an automated assistant (via a spoken or typed dialog session) to interact with the third party web site without requiring the user to visually interact with the third party web site directly and without requiring the third party to implement their own third party agent.

Analyzing web pages to facilitate automatic navigation

Techniques described herein enable a user to interact with an automated assistant and obtain relevant output from the automated assistant without requiring arduous typed input to be provided by the user and/or without requiring the user to provide spoken input that could cause privacy concerns (e.g., if other individuals are nearby). The assistant application can operate in multiple different image conversation modes in which the assistant application is responsive to various objects in a field of view of the camera. The image conversation modes can be suggested to the user when a particular object is detected in the field of view of the camera. When the user selects an image conversation mode, the assistant application can thereafter provide output, for presentation, that is based on the selected image conversation mode and that is based on object(s) captured by image(s) of the camera.

Systems, methods, and apparatus for image-responsive automated assistants

Techniques are described herein for automated assistants that search various alternative corpora for information. In various implementations, a method may include receiving, by an automated assistant via an input component of a first client device, a free form input, wherein the free form input includes a request for specific information; searching a general purpose corpus of online documents to obtain a first set of candidate response(s) to the request for specific information; searching a user-specific corpus of active document(s) to obtain a second set of candidate response(s) to the request for specific information; comparing the first and second sets of candidate responses; based on the comparing, selecting a given response to the request for specific information from the first or second set; and providing, by the automated assistant, output indicative of the given response.

Obtaining responsive information from multiple corpora

The present disclosure relates to user-selected metadata related to images captured by a camera of a client device. User-selected metadata may include contextual information and/or information provided by a user when the images are captured. In various implementations, a free form input may be received at a first client device of one or more client devices operated by a user. A task request may be recognized from the free form input, and it may be determined that the task request includes a request to store metadata related to one or more images captured by a camera of the first client device. The metadata may be selected based on content of the task request. The metadata may then be stored, e.g., in association with one or more images captured by the camera, in computer-readable media. The computer-readable media may be searchable by the metadata.

Storing metadata related to captured images

Growing community of inventors

Zurich, Switzerland

Gökhan Hasan Bakir