Google Inc.

Marcin M Nowak-Przygodzki

Behshad Behzadi

Mugurel Ionut Andreica

Gokhan Hasan Bakir

Joseph Lange

Gökhan Hasan Bakir

Aurelien Boffy

Vladimir Vuskovic

Evgeny A Cherepanov

Nils Grimsmo

Matthias Heiler

Fredrik Bergenlid

Karoly Csalogany

Alessandro Agostini

Andrei Giurgiu

Gabor Simko

Tak Keung Cheng

Nathan David Howard

Radu Voroneanu

Gökhan Bakir

Michal Jastrzebski

Sharon Stovezky

Zaheed Sabur

Daniel Cotting

Lucas Mirelmann

Denis Burakov

Sergey Nazarov

Michael Golikov

Steve Cheng

Mario Bertschler

Bohdan Vlasyuk

Mugurel-Ionut Andreica

Jan Lamecki

Matthew Stokes

Andre Elisseeff

Dana Ritter

Torsten Marek

João Paulo Pagaime Da Silva

Mathias Carlen

Lukasz Suder

Ernest Galbrun

Ibrahim Badr

Gabriel Taubman

Ilya Firman

Adam Coimbra

James Stout

Yariv Adan

Thomas Weedon Hume

Abhanshu Sharma

Aneto Okonkwo

Jindong Chen

Jacek Szmigiel

Kyle Pedersen

Aurélien Boffy

Tatsiana Sakhar

Adrian-Marius Dumitran

Marius Dumitran

Ragnar Groot Koerkamp

Jonathan Small

Reed Enger

Vishwath Mohan

Yunfan Jin

Gilles Baechler

Implementations set forth herein relate to a system that employs an automated assistant to further interactions between a user and another application, which can provide the automated assistant with permission to initialize relevant application actions simultaneous to the user interacting with the other application. Furthermore, the system can allow the automated assistant to initialize actions of different applications, despite being actively operating a particular application. Available actions can be gleaned by the automated assistant using various application-specific schemas, which can be compared with incoming requests from a user to the automated assistant. Additional data, such as context and historical interactions, can also be used to rank and identify a suitable application action to be initialized via the automated assistant.

Initializing non-assistant background actions, via an automated assistant, while accessing a non-assistant application

Implementations set forth herein relate to an automated assistant that can operate as an interface between a user and a separate application to search application content of the separate application. The automated assistant can interact with existing search filter features of another application and can also adapt in circumstances when certain filter parameters are not directly controllable at a search interface of the application. For instance, when a user requests that a search operation be performed using certain terms, those terms may refer to content filters that may not be available at a search interface of the application. However, the automated assistant can generate an assistant input based on those content filters in order to ensure that any resulting search results will be filtered accordingly. The assistant input can then be submitted into a search field of the application and a search operation can be executed.

Automated assistant for introducing or controlling search filter parameters at a separate application

Techniques are described herein for delegation of request fulfillment, by an assistant, to other devices. A method includes: receiving, by a first device, a request from a first user; identifying, based on the request from the first user, (i) an action corresponding to the request and (ii) a first parameter corresponding to the action; determining that fulfillment of the action is to be delegated to a device other than the first device; in response: selecting, as the device other than the first device, a second device on which an application corresponding to the action is installed; identifying, by the first device, based on the first parameter and information associated with an account of the first user, a first disambiguated parameter corresponding to the action; and sending, to the second device, a command that specifies the action and the first disambiguated parameter, to cause the second device to fulfill the action.

Assistant delegation of request fulfillment to other devices

Methods, systems, and apparatus, for associating data items with objects depicted in an image and search the associated data items based on object detected in images that are subsequently processed and presenting the data items associated with an object when the object is identified in the subsequently processed image.

Association of data items and objects

Generating and/or utilizing image shortcuts that cause one or more corresponding computer actions to be performed in response to determining that one or more features are present in image(s) from a camera of a computing device of a user (e.g., present in a real-time image feed from the camera). An image shortcut can be generated in response to user interface input, such as a spoken command. For example, the user interface input can direct the automated assistant to perform one or more actions in response to object(s) having certain feature(s) being present in a field of view of the camera. Subsequently, when the user directs their camera at object(s) having such feature(s), the assistant application can cause the action(s) to be automatically performed. For example, the assistant application can cause data to be presented and/or can control a remote device in accordance with the image shortcut.

Systems, methods, and apparatus for providing image shortcuts for an assistant application

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating subqueries from a query. In one aspect, a method includes obtaining a query, generating a set of two subqueries from the query, where the set includes a first subquery and a second subquery, determining a quality score for the set of two subqueries, determining whether the quality score for the set of two subqueries satisfies a quality threshold, and in response to determining that the quality score for the set of two subqueries satisfies the quality threshold, providing a first response to the first subquery that is responsive to a first operation that receives the first subquery as input and providing a second response to the second subquery that is responsive to a second operation that receives the second subquery as input.

Subquery generation from a query

A computer-implemented method is described. The method includes a computing system receiving a first voice input from a user for generating a bookmark. Responsive to receiving the voice input, the computing system obtains a resource identifier associated with a resource page. The method includes the computing system storing the resource identifier as a bookmark in a bookmark index that is accessible in response to a request from a user device. The method further includes the computing system using the resource identifier, stored as the bookmark, to access the resource page. The computing system uses the resource identifier to access the resource page based on the request from the user device. The request is generated by the user device in response to the user device receiving a second voice input from the user.

Contextual deep bookmarking

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving a query provided by a user and comprising one or more terms. Obtaining context data based on at least a portion of a first resource displayed to the user at a time that the query is received. Obtaining a revised query that is based on the query and the context data. Receiving a plurality of search results responsive to the revised query. Automatically, selecting a search result that represents a second resource from the plurality of search results, and providing the second resource for display to the user.

Assistive browsing using context

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for detecting a continued conversation are disclosed. In one aspect, a method includes the actions of receiving first audio data of a first utterance. The actions further include obtaining a first transcription of the first utterance. The actions further include receiving second audio data of a second utterance. The actions further include obtaining a second transcription of the second utterance. The actions further include determining whether the second utterance includes a query directed to a query processing system based on analysis of the second transcription and the first transcription or a response to the first query. The actions further include configuring the data routing component to provide the second transcription of the second utterance to the query processing system as a second query or bypass routing the second transcription.

Detecting conversations with computing devices

Techniques described herein enable a user to interact with an automated assistant and obtain relevant output from the automated assistant without requiring arduous typed input to be provided by the user and/or without requiring the user to provide spoken input that could cause privacy concerns (e.g., if other individuals are nearby). The assistant application can operate in multiple different image conversation modes in which the assistant application is responsive to various objects in a field of view of the camera. The image conversation modes can be suggested to the user when a particular object is detected in the field of view of the camera. When the user selects an image conversation mode, the assistant application can thereafter provide output, for presentation, that is based on the selected image conversation mode and that is based on object(s) captured by image(s) of the camera.

Systems, methods, and apparatus for image-responsive automated assistants

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining query refinements using search data. In one aspect, a method includes receiving a first query and a second query each comprising one or more n-grams for a user session, determining a first set of query refinements for the first query, determining a second set of query refinements from the first set of query refinements, each query refinement in the second set of query refinements including at least one n-gram that is similar to an n-gram from the first query and at least on n-gram that is similar to an n-gram from the second query, scoring each query refinement in the second set of query refinements, selecting a third query from a group consisting of the second set of query refinements and the second query, and providing the third query as input to a search operation.

Query refinements using search data

Implementations are set forth herein for creating an order of execution for actions that were requested by a user, via a spoken utterance to an automated assistant. The order of execution for the requested actions can be based on how each requested action can, or is predicted to, affect other requested actions. In some implementations, an order of execution for a series of actions can be determined based on an output of a machine learning model, such as a model that has been trained according to supervised learning. A particular order of execution can be selected to mitigate waste of processing, memory, and network resources—at least relative to other possible orders of execution. Using interaction data that characterizes past performances of automated assistants, certain orders of execution can be adapted over time, thereby allowing the automated assistant to learn from past interactions with one or more users.

Orchestrating execution of a series of actions requested to be performed via an automated assistant

Implementations are directed to determining, based on a submitted query that is a compound query, that a set of multiple sub-queries are collectively an appropriate interpretation of the compound query. Those implementations are further directed to providing, in response to such a determination, a corresponding command for each of the sub-queries of the determined set. Each of the commands is to a corresponding agent (of one or more agents), and causes the agent to generate and provide corresponding responsive content. Those implementations are further directed to causing content to be rendered in response to the submitted query, where the content is based on the corresponding responsive content received in response to the commands.

Determining responsive content for a compound query based on a set of generated sub-queries

Growing community of inventors

Zurich, Switzerland

Marcin M Nowak-Przygodzki