Google Inc.

International Business Machines Corporation

Keun Soo Yim

Iliyan Batanov Malchev

Zhitu Chen

Arun Kwangil Iyengar

Liana Liyow Fong

Seetharami R Seelam

Ilya Firman

John Lewars

Umesh Patil

Brendan G Lim

Kyung Yul Lim

Gabriel Taubman

Pratyus Patnaik

Adam Coimbra

Mugurel Ionut Andreica

Svetoslav Ganov

Bogdan Caprita

Cliff Kuang

Maya Ben Ari

Mikhail Reutov

Robert Berry

Michael Oh

Kalyana Ram Desineni

Ravi Shah

Implementations relate to triggering suggestion(s) for a document that is at least partially displayed by a content access application at a user interface of a client computing device. The suggestions(s) can be triggered when one or more triggering conditions that provide when to trigger the suggestion(s) are satisfied. The one or more triggering conditions can include, for example, a coordinate condition, a DOM node condition, and/or a temporal condition.

Mitigating latency and/or resource usage in triggering actionable suggestions related to rendered content

The present disclosure is directed to automating user interactions in an application. In particular, the computing device can determine, based on the input data, an intent associated with performing an action on the computing device. Additionally, the computing device can invoke, using an application-level assistant component configured to provide an application-level interface on the computing device, execution of an application. Moreover, the computing device can transmit, to a system-level assistant component configured to provide a system-level interface with the computing device, instructions to interface with an actionable component of the application. The instructions can be transmitted using a trusted intermediary component configured to facilitate an interface between the system-level assistant component and the application-level assistant component. Furthermore, the computing device can engage, using the system-level assistant component, the actionable component of the application to perform the action.

Human-in-the-loop voice automation system

Implementations include recognizing entities in a user query received from a client device and determining whether any of the recognized entities is a personal entity using an entity database storing personal entities donated by applications installed at the client device. Application(s) or application action(s) can be determined based on entity information of the recognized entities, and suggestions can be generated to suggest the determined application(s) (or the determined application action(s)). For an application (or application action) determined based on a recognized entity that is determined as a personal entity, a suggestion for the application (or application action) can be boosted for display to a user of the client device.

Rendering suggestion for searching entity within application in response to determining the entity is associated with the application

Implementations determine log-in information indicating whether a user of a first application is logged into a target application and/or a website of the target application. In response to the log-in information indicating that the user is logged into the target application but not logged into the website of the target application, a first selectable element can be displayed at a user interface of the first application to receive user input that causes the target application to be opened. In response to the log-in information indicating that the user is not logged into the target application but is logged into the website of the target application, a second selectable element can be displayed at the first application to receive user input that leads to the website of the target application, or the first selectable element can be assigned a deeplink that leads to the website of the target application.

Launching determination based on login status

Implementations set forth herein relate to selectively relying on additional suggestion model(s) when generating selectable suggestions, while also maintaining access to a default suggestion model. The selectable suggestions can be generated using one or more additional multi-domain machine learning (ML) models, which can be optionally available to the client application, regardless of whether a default suggestion model remains useful for generating suitable suggestions. In some implementations, as the client application employs various additional multi-domain ML models, a particular model can be identified as improving suggestions for the client application, at least based on user feedback and/or other data. The particular model can then be selected to replace and/or supplement the default suggestion model, in order to provide more accurate suggestions that, when selected, initialize actions that can preserve time and computational resources.

Generating a selectable suggestion using a provisional machine learning model when use of a default suggestion model is inconsequential

Implementations relate to processing media content, and/or associated metadata, to classify the media content into a first category, of a plurality of predefined categories. Versions of those implementations further relate to extracting target content from the media content; generating, based on the extracted target content, an action that corresponds to an application; and generating, based on the generated action, a selectable suggestion including a textual portion that describes the action. Some of those versions further relate to causing the selectable suggestion to be displayed at a display of a client device, along with rendering of the media content. The selectable suggestion, when selected, causes the application to perform the action. The target content can be extracted based on the first category and can be extracted based on the first category in response to the media content being classified into the first category.

Actionable suggestions for media content

A method of encoding and decoding user interface interactions includes receiving a query from a user captured by an assistant-enabled device associated with the user. The query includes a user intent for interacting with an application. The method includes obtaining, for the application, a set of categorized actions. Each categorized action of the set of categorized actions is associated with one or more parameters and represents a high-level description of the user intent of the user interacting with the application. For each respective categorized action of the set of categorized actions, the method includes selecting a respective script associated with the respective categorized action that performs one or more low-level interactions with the application and executing the respective script to represent the user intent for interacting with the application.

Encoding/decoding user interface interactions

Implementations set forth herein relate to providing selectable autofill suggestions, which correspond to application actions that are at least partially fulfilled using server command data—prior to a user selecting a particular selectable autofill suggestion. Proactively fulfilling command data in this way mitigates latency between user selection of a suggestion and fulfillment of a particular action. Initially, a partial input can be processed to generate autofill suggestions, which can be communicated to a server device for further processing. The autofill suggestions can also be rendered for selection at a touch display interface, thereby allowing a user to select one of the autofill suggestions. As command fulfillment data is provided by the server, the command fulfillment data can be available to a corresponding application(s) in order that any corresponding actions can be at least partially fulfilled prior to user selection.

Fulfillment of actionable requests ahead of a user selecting a particular autocomplete suggestion for completing a current user input

Implementations set forth herein relate to an automated assistant that can adapt to circumstances in which a user may invoke an automated assistant with an intention of interacting with the automated assistant via a non-default interface. For example, in some instances, a user may invoke an automated assistant by selecting a selectable GUI element. In response, the automated assistant can determine that, in the current context, spoken utterances may not be suitable for providing to the automated assistant. Based on this determination, the automated assistant can cause a keyboard interface to be rendered and/or initialized for receiving typed inputs from the user. Should the user subsequently change contexts, the automated assistant can determine that voice input is now suitable for user input and initialize an audio interface in response to the user providing an invocation input in the subsequent context.

Selectively rendering a keyboard interface in response to an assistant invocation in certain circumstances

Implementations set forth herein relate to providing suggestions for application services using tree data structures provided by various application sources. A native data structure, created by an assistant application or other related application, can be present in a local memory and, depending on the status of this native data structure, can be utilized to generate deep links to be rendered within the search interface. Selection of a particular deep link can initialize particular operations and/or routines at one or more client or remote applications. When the native structure is unavailable, or does not satisfy certain criteria, other actions can be initiated for fetching data from remote sources or utilizing other locally available application data structures. In some implementations, graphical user interface elements indicating operation statuses can be rendered based on the available data structure(s), thereby aiding in selecting certain services across multiple applications.

Disparate sourcing of tree data structures for searching and suggesting application services

Implementations are described herein for recommending actions based on entity or entity type. In various implementations, a partial free-form natural language input may be received from a user at an input component of a computing device. The partial free-form natural language input may identify an entity without identifying a responsive action and may be directed by the user to an automated assistant that operates at least in part on the computing device. The partial free-form natural language input may be analyzed to identify the entity. Based on the identified entity, a plurality or superset of candidate responsive actions may be identified, filtered, and/or ranked based on one or more signals. The automated assistant may then provide output that recommends one or more of the candidate responsive actions based on the ranking and/or filtering.

Recommending action(s) based on entity or entity type

Implementations set forth herein relate to an automated assistant that allows third party applications to inject dependencies to leverage automated assistant functions. Furthermore, enabling such dependency injections can allow third party applications to preserve privacy of any application content that is used during execution of automated assistant functions. In some implementations, a third party application can initialize a function with an assistant dependency using parameters that are tagged as private. Initializing a function in such as a way can allow private content communicated between the third party application and the automated assistant to be abstracted for security purposes. The abstracted content can thereafter be communicated to a remote server—such as a server hosting an extensively trained machine learning model. Intelligent output provided by the server can then be incorporated into one or more processes of the third party application without comprising security.

Automated assistant architecture for preserving privacy of application content

In various implementations described herein, a partial free-form natural language input may be received from a user at an input component of a computing device. The partial free-form natural language input may identify an entity without identifying a responsive action and may be directed by the user to an automated assistant that operates at least in part on the computing device. The partial free-form natural language input may be analyzed to identify the entity. Based on the identified entity, a plurality or superset of candidate responsive actions may be identified, filtered, and/or ranked based on one or more signals. The automated assistant may then provide output that recommends one or more of the candidate responsive actions based on the ranking and/or filtering.

Growing community of inventors

San Jose, CA, United States of America

Keun Soo Yim