Google Inc.

Daniel Cotting

Behshad Behzadi

Zaheed Sabur

Lucas Mirelmann

Denis Burakov

Sergey Nazarov

Michael Golikov

Steve Cheng

Mario Bertschler

Bohdan Vlasyuk

Boris Bluntschli

Bryan Christopher Horling

Daniel Joseph Filip

Luv Kothari

Lan Huo

Marcin M Nowak-Przygodzki

Jonathan David Lee

Okan Kolak

Vinh Quoc Ly

Adrian Zumbrunnen

Mugurel Ionut Andreica

Yan Zhong

Roland Peter Kehl

Radu Voroneanu

Lucia Terrenghi

Daniel Raynaud

Marie Patriarche Bledowski

Masoud Loghmani

Andrea Martinelli

Bernhard Rudolf Suter

Dan Filimon

Wojciech Stanislaw Smietanka

Elizabeth Hamon Reid

Avni Shah

Hylke Niekele Buisman

Krzysztof Sikora

Nicolas Dumazet

Martin Stefcek

Implementations set forth herein relate to a system that employs an automated assistant to further interactions between a user and another application, which can provide the automated assistant with permission to initialize relevant application actions simultaneous to the user interacting with the other application. Furthermore, the system can allow the automated assistant to initialize actions of different applications, despite being actively operating a particular application. Available actions can be gleaned by the automated assistant using various application-specific schemas, which can be compared with incoming requests from a user to the automated assistant. Additional data, such as context and historical interactions, can also be used to rank and identify a suitable application action to be initialized via the automated assistant.

Initializing non-assistant background actions, via an automated assistant, while accessing a non-assistant application

Implementations can reduce the time required to obtain responses from an automated assistant by, for example, obviating the need to provide an explicit invocation to the automated assistant, such as by saying a hot-word/phrase or performing a specific user input, prior to speaking a command or query. In addition, the automated assistant can optionally receive, understand, and/or respond to the command or query without communicating with a server, thereby further reducing the time in which a response can be provided. Implementations only selectively initiate on-device speech recognition responsive to determining one or more condition(s) are satisfied. Further, in some implementations, on-device NLU, on-device fulfillment, and/or resulting execution occur only responsive to determining, based on recognized text form the on-device speech recognition, that such further processing should occur. Thus, through selective activation of on-device speech processing, and/or selective activation of on-device NLU and/or on-device fulfillment, various client device resources are conserved.

Selectively activating on-device speech recognition, and using recognized text in selectively activating on-device NLU and/or on-device fulfillment

Implementations can reduce the time required to obtain responses from an automated assistant through proactive caching, locally at a client device, of proactive assistant cache entries—and through on-device utilization of the proactive assistant cache entries. Different proactive cache entries can be provided to different client devices, and various implementations relate to technique(s) utilized in determining which proactive cache entries to provide to which client devices. In some of those implementations, in determining which proactive cache entries to provide (proactively or in response to a request) to a given client device, a remote system selects, from a superset of candidate proactive cache entries, a subset of the cache entries for providing to the given client device.

Proactive caching of assistant action content to enable resolution of spoken or typed utterances

This disclosure relates to digital video analysis. In one aspect, a method includes providing a user interface that enables a user of the computing system to select one or more seed video groups and one or more keywords, wherein each seed video group comprises one or more videos. The user interface is updated to provide candidate video groups selected based on the one or more seed video groups and the one or more keywords and, for each candidate video group, a first user interface control that enables the user to refine the set of candidate video groups to include video groups classified as being similar to the candidate video group. Data indicating user interaction with a given first user interface control for a first candidate video group is received. The user interface is updated to provide an updated set of candidate video groups.

User interfaces for refining video group packages

This disclosure relates to digital video analysis. In one aspect, a method includes receiving data indicating one or more seed video groups that each include one or more seed videos. Data indicating one or more keywords is received. A set of candidate video groups that each include one or more candidate videos is identifier. For each candidate video group in the set of candidate video groups a co-interaction score and a topicality score are determined. A subset of the candidate videos groups is selected based on the co-interaction score and the topicality score of each candidate video group. Data indicating the subset of candidate video groups is provided for presentation.

Digital video analysis

Implementations set forth herein allow a user to access a first application in a foreground of a graphical interface, and simultaneously employ an automated assistant to respond to notifications arising from a second application. The user can provide an input, such as a spoken utterance, while viewing the first application in the foreground in order to respond to notifications from the second application without performing certain intervening steps that can arise under certain circumstances. Such intervening steps can include providing a user confirmation, which can be bypassed, and/or time-limited according to a timer, which can be displayed in response to the user providing a responsive input directed at the notification. A period for the timer can be set according to one or more characteristics that are associated with the notification, the user, and/or any other information that can be associated with the user receiving the notification.

Dynamically delaying execution of automated assistant actions and/or background application requests

Implementations relate to receiving natural language input that requests an automated assistant to provide information and processing the natural language input to identify the requested information and to identify one or more predicted actions. Those implementations further cause a computing device, at which the natural language input is received, to render the requested information and the one or more predicted actions in response to the natural language input. Yet further, those implementations, in response to the user confirming a rendered predicted action, cause the automated assistant to initialize the predicted action.

Performing subtask(s) for a predicted action in response to a separate user interaction with an automated assistant prior to performance of the predicted action

Implementations set forth herein relate to an automated assistant that can be invoked while a user is interfacing with a foreground application in order to retrieve data from one or more different applications, and then provide the retrieved data to the foreground application. A user can invoke the automated assistant while operating the foreground application by providing a spoken utterance, and the automated assistant can select one or more other applications to query based on content of the spoken utterance. Application data collected by the automated assistant from the one or more other applications can then be used to provide an input to the foreground application. In this way, the user can bypass switching between applications in the foreground in order to retrieve data that has been generated by other applications.

Automated assistant for generating, in response to a request from a user, application input content using application data from other sources

Implementations herein relate to pre-caching data, corresponding to predicted interactions between a user and an automated assistant, using data characterizing previous interactions between the user and the automated assistant. An interaction can be predicted based on details of a current interaction between the user and an automated assistant. One or more predicted interactions can be initialized, and/or any corresponding data pre-cached, prior to the user commanding the automated assistant in furtherance of the predicted interaction. Interaction predictions can be generated using a user-parameterized machine learning model, which can be used when processing input(s) that characterize a recent user interaction with the automated assistant. The predicted interaction(s) can include action(s) to be performed by third-party application(s).

Implementations can reduce the time required to obtain responses from an automated assistant through proactive caching, locally at a client device, of proactive assistant cache entries&#x2014;and through on-device utilization of the proactive assistant cache entries. Different proactive cache entries can be provided to different client devices, and various implementations relate to technique(s) utilized in determining which proactive cache entries to provide to which client devices. In some of those implementations, in determining which proactive cache entries to provide (proactively or in response to a request) to a given client device, a remote system selects, from a superset of candidate proactive cache entries, a subset of the cache entries for providing to the given client device.

Growing community of inventors

Islisberg, Switzerland

Daniel Cotting