Google Inc.

Cliff Kuang

Jonathan David Lee

Andy Gugel

Jesse Kaczmarek

Adam Coimbra

Radu Voroneanu

Mugurel-Ionut Andreica

Andre Elisseeff

Dana Ritter

Diana Avram

Alice Liang

Gabriel Taubman

Keun Soo Yim

Ilya Firman

Mugurel Ionut Andreica

Ian Baker

Svetoslav Ganov

Mikhail Reutov

Kevin Gaunt

Robert Berry

Michael Oh

Sneha Ashok

Deepak Goyal

Anton Berezin

Andrea D'olimpio

Florian Hasler

Ricardo Bruno Augusto Henriques

Kalyana Ram Desineni

Katrina Egert-Pecot

Abraham Wallin

Caleb Misclevitz

Karan Patel

Ryan Allen

Simon Sok

Sean Joseph Klassen

Stacy Cronin

Joshua Lance Leviste Principe

Kyunghoon Lee

The present disclosure is directed to automating user interactions in an application. In particular, the computing device can determine, based on the input data, an intent associated with performing an action on the computing device. Additionally, the computing device can invoke, using an application-level assistant component configured to provide an application-level interface on the computing device, execution of an application. Moreover, the computing device can transmit, to a system-level assistant component configured to provide a system-level interface with the computing device, instructions to interface with an actionable component of the application. The instructions can be transmitted using a trusted intermediary component configured to facilitate an interface between the system-level assistant component and the application-level assistant component. Furthermore, the computing device can engage, using the system-level assistant component, the actionable component of the application to perform the action.

Human-in-the-loop voice automation system

Example embodiments of the present disclosure provide for an example method including obtaining component data associated with a plurality of components. The example method includes generating an index of the component data and obtaining user input data indicative of a request. The example method includes processing the user input data to determine an intent associated with the request and obtaining, based on the intent, data indicative of one or more components from the index. The example method includes determining a rank for at least one respective component of the one or more components. The example method includes updating a user interface to display the one or more components, wherein the components are displayed based at least in part on the rank of the respective component. The example method can include generating modular spaces composed of a plurality of components to generate composite application interfaces which can include cross-application functionalities.

Retrieval and composition of modular spaces

Implementations set forth herein relate to an automated assistant that can proactively identify and complete tasks that may be associated with an activity with which a user has scheduled the automated assistant to assist with. The tasks can be identified and completed prior to a time that the user has scheduled the automated assistant to assist the user, thereby eliminating certain manual tasks the user may otherwise perform at the scheduled time. When the activity involves communicating with a separate entity, such as another person and/or organization, the automated assistant can initialize communication with the entity prior to the scheduled time that the user requested assistance. A customized GUI can be rendered at an assistant-enabled device to provide the user with an ongoing status of completing various tasks associated with scheduled activity.

Assistant adaptation of graphical user interface to guide interaction with user in fulfilling user request

Implementations relate to identifying actions performed by a user while the user is interacting with multiple applications and providing a routine suggestion to the user based on the identified actions. While a user is interacting with one or more applications, the user actions are determined. The user actions are compared to one or more template routines and a template routine is selected that matches the actions of the user and provided to the user as a suggested routine. The suggested routine can be implemented by an automated assistant to perform the actions of the template by providing a corresponding command.

Automatically suggesting routines based on detected user actions via multiple applications

Implementations relate to determining a rendering type for an application that is executing automatically. Based on user interactions with an application that is associated with specified input from the user while the user is interacting with the application, a confidence metric is generated for each specified input and a rendering type is determined based on the confidence metrics. Subsequently, when the user requests that a sequence of actions be performed, the application will be displayed according to the rendering type.

Automatically executing application routines with user inputs

Implementations set forth herein relate to providing notifications regarding events that may interrupt content being rendered at an interface. The notifications can be preemptive and/or can indicate a predicted time and/or source for the events. The event can be, for example, a person attempting to contact a user who is viewing content at a display interface. The person can be associated with a food delivery that has been ordered by the user via a delivery application. An application, such an automated assistant application, can predict when the person is expected to arrive with the food delivery, and generate a notification ahead of the person arriving. In some implementations, the notification can be rendered at a scrubber user interface (UI) at a location corresponding to the time that the food delivery is expected to arrive, thereby putting the user on notice of when the streaming content may be interrupted.

Generating notifications that provide context for predicted content interruptions

Some implementations relate to an automated assistant that can provide assistant suggestions based on content that the user has requested for the automated assistant to render at a computing device. In some instances, an assistant suggestion can correspond to a portion of the content that is not being rendered at the computing device but that is part of the content that the user is accessing. In other instances, an assistant suggestion can be based on data that can indicate how one or more users may have interacted with their automated assistant when viewing the content. Therefore, an assistant suggestion can correspond to an assistant operation that may have been initialized by another user who had previously viewed the content. In some implementations, an arrangement of the assistant suggestions can be modified according to which part of the content a user is viewing and/or user distance.

Dynamic assistant suggestions during assistant browsing

Growing community of inventors

San Francisco, CA, United States of America

Cliff Kuang