Google Inc.

Mugurel Ionut Andreica

Marcin M Nowak-Przygodzki

Gokhan Hasan Bakir

Joseph Lange

Behshad Behzadi

Vladimir Vuskovic

Tak Keung Cheng

Zaheed Sabur

Lucas Mirelmann

Daniel Cotting

Denis Burakov

Sergey Nazarov

Michael Golikov

Steve Cheng

Mario Bertschler

Bohdan Vlasyuk

Radu Voroneanu

Sharon Stovezky

Ibrahim Badr

Gabriel Taubman

Keun Soo Yim

Ilya Firman

Adam Coimbra

Gökhan Hasan Bakir

Svetoslav Ganov

Cliff Kuang

Mikhail Reutov

Michal Jastrzebski

Michael Oh

Robert Berry

Kalyana Ram Desineni

Tatsiana Sakhar

Adrian-Marius Dumitran

Jonathan Small

Implementations set forth herein relate to a system that employs an automated assistant to further interactions between a user and another application, which can provide the automated assistant with permission to initialize relevant application actions simultaneous to the user interacting with the other application. Furthermore, the system can allow the automated assistant to initialize actions of different applications, despite being actively operating a particular application. Available actions can be gleaned by the automated assistant using various application-specific schemas, which can be compared with incoming requests from a user to the automated assistant. Additional data, such as context and historical interactions, can also be used to rank and identify a suitable application action to be initialized via the automated assistant.

Initializing non-assistant background actions, via an automated assistant, while accessing a non-assistant application

The present disclosure is directed to automating user interactions in an application. In particular, the computing device can determine, based on the input data, an intent associated with performing an action on the computing device. Additionally, the computing device can invoke, using an application-level assistant component configured to provide an application-level interface on the computing device, execution of an application. Moreover, the computing device can transmit, to a system-level assistant component configured to provide a system-level interface with the computing device, instructions to interface with an actionable component of the application. The instructions can be transmitted using a trusted intermediary component configured to facilitate an interface between the system-level assistant component and the application-level assistant component. Furthermore, the computing device can engage, using the system-level assistant component, the actionable component of the application to perform the action.

Human-in-the-loop voice automation system

A computer-implemented method is described. The method includes a computing system receiving a first voice input from a user for generating a bookmark. Responsive to receiving the voice input, the computing system obtains a resource identifier associated with a resource page. The method includes the computing system storing the resource identifier as a bookmark in a bookmark index that is accessible in response to a request from a user device. The method further includes the computing system using the resource identifier, stored as the bookmark, to access the resource page. The computing system uses the resource identifier to access the resource page based on the request from the user device. The request is generated by the user device in response to the user device receiving a second voice input from the user.

Contextual deep bookmarking

Implementations are set forth herein for creating an order of execution for actions that were requested by a user, via a spoken utterance to an automated assistant. The order of execution for the requested actions can be based on how each requested action can, or is predicted to, affect other requested actions. In some implementations, an order of execution for a series of actions can be determined based on an output of a machine learning model, such as a model that has been trained according to supervised learning. A particular order of execution can be selected to mitigate waste of processing, memory, and network resources—at least relative to other possible orders of execution. Using interaction data that characterizes past performances of automated assistants, certain orders of execution can be adapted over time, thereby allowing the automated assistant to learn from past interactions with one or more users.

Orchestrating execution of a series of actions requested to be performed via an automated assistant

Implementations are directed to determining, based on a submitted query that is a compound query, that a set of multiple sub-queries are collectively an appropriate interpretation of the compound query. Those implementations are further directed to providing, in response to such a determination, a corresponding command for each of the sub-queries of the determined set. Each of the commands is to a corresponding agent (of one or more agents), and causes the agent to generate and provide corresponding responsive content. Those implementations are further directed to causing content to be rendered in response to the submitted query, where the content is based on the corresponding responsive content received in response to the commands.

Determining responsive content for a compound query based on a set of generated sub-queries

Implementations are set forth herein for creating an order of execution for actions that were requested by a user, via a spoken utterance to an automated assistant. The order of execution for the requested actions can be based on how each requested action can, or is predicted to, affect other requested actions. In some implementations, an order of execution for a series of actions can be determined based on an output of a machine learning model, such as a model that has been trained according to supervised learning. A particular order of execution can be selected to mitigate waste of processing, memory, and network resources&#x2014;at least relative to other possible orders of execution. Using interaction data that characterizes past performances of automated assistants, certain orders of execution can be adapted over time, thereby allowing the automated assistant to learn from past interactions with one or more users.

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for receiving a first input associated with a bookmark and, in response to receiving the first input, obtaining context data for deriving annotations for the bookmark. An annotation engine of a system obtains multiple annotations that are derived using information included in the context data and a transcription of the first input associated with the bookmark. The annotation engine generates an importance value for each annotation. The importance value characterizes a relationship between each annotation and at least one of: the information included in the context data; or the transcription of the first input. The annotation engine generates a set of annotations for retrieving the bookmark. Each annotation in the set of annotations is generated from the obtained multiple annotations and each annotation has an importance value that exceeds a threshold value.

Annotation and retrieval of personal bookmarks

Techniques are described herein for automated assistants that search various alternative corpora for information. In various implementations, a method may include receiving, by an automated assistant via an input component of a first client device, a free form input, wherein the free form input includes a request for specific information; searching a general purpose corpus of online documents to obtain a first set of candidate response(s) to the request for specific information; searching a user-specific corpus of active document(s) to obtain a second set of candidate response(s) to the request for specific information; comparing the first and second sets of candidate responses; based on the comparing, selecting a given response to the request for specific information from the first or second set; and providing, by the automated assistant, output indicative of the given response.

Obtaining responsive information from multiple corpora

In some implementations, a query that includes a sequence of terms is obtained, the query is mapped, based on the sequence of the terms, to a dependency tree that represents dependencies among the terms in the query, an entity type that corresponds to an entity sought by the query is determined based on a term represented by a root of the dependency tree, a particular entity is identified based on both the entity type and a relevance of the entity to the terms in the query, and a response to the query is provided based on the particular entity that is identified.

Answering entity-seeking queries

Growing community of inventors

Adliswil, Switzerland

Mugurel Ionut Andreica