Google Inc.

Joseph Lange

Marcin M Nowak-Przygodzki

Vladimir Vuskovic

Mugurel Ionut Andreica

Jacek Szmigiel

Behshad Behzadi

Sharon Stovezky

Michal Pryt

Olivier Siegenthaler

Asier Aguirre

Gabriel Taubman

Ilya Firman

Adam Coimbra

James Stout

Thomas Weedon Hume

Abhanshu Sharma

Jindong Chen

Gökhan Bakir

Andrei Giurgiu

Mugurel-Ionut Andreica

Kyle Pedersen

Henry Scott Dlhopolsky

Reed Enger

Vishwath Mohan

Yunfan Jin

Gilles Baechler

Implementations set forth herein relate to an automated assistant that can operate as an interface between a user and a separate application to search application content of the separate application. The automated assistant can interact with existing search filter features of another application and can also adapt in circumstances when certain filter parameters are not directly controllable at a search interface of the application. For instance, when a user requests that a search operation be performed using certain terms, those terms may refer to content filters that may not be available at a search interface of the application. However, the automated assistant can generate an assistant input based on those content filters in order to ensure that any resulting search results will be filtered accordingly. The assistant input can then be submitted into a search field of the application and a search operation can be executed.

Automated assistant for introducing or controlling search filter parameters at a separate application

Techniques are described herein for delegation of request fulfillment, by an assistant, to other devices. A method includes: receiving, by a first device, a request from a first user; identifying, based on the request from the first user, (i) an action corresponding to the request and (ii) a first parameter corresponding to the action; determining that fulfillment of the action is to be delegated to a device other than the first device; in response: selecting, as the device other than the first device, a second device on which an application corresponding to the action is installed; identifying, by the first device, based on the first parameter and information associated with an account of the first user, a first disambiguated parameter corresponding to the action; and sending, to the second device, a command that specifies the action and the first disambiguated parameter, to cause the second device to fulfill the action.

Assistant delegation of request fulfillment to other devices

Implementations set forth herein relate to an automated assistant that can control graphical user interface (GUI) elements via voice input using natural language understanding of GUI content in order to resolve ambiguity and allow for condensed GUI voice input requests. When a user is accessing an application that is rendering various GUI elements at a display interface, the automated assistant can operate to process actionable data corresponding to the GUI elements. The actionable data can be processed in order to determine a correspondence between GUI voice input requests to the automated assistant and at least one of the GUI elements rendered at the display interface. When a particular spoken utterance from the user is determined to correspond to multiple GUI elements, an indication of ambiguity can be rendered at the display interface in order to encourage the user to provide a more specific spoken utterance.

Condensed spoken utterances for automated assistant control of an intricate application GUI

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating subqueries from a query. In one aspect, a method includes obtaining a query, generating a set of two subqueries from the query, where the set includes a first subquery and a second subquery, determining a quality score for the set of two subqueries, determining whether the quality score for the set of two subqueries satisfies a quality threshold, and in response to determining that the quality score for the set of two subqueries satisfies the quality threshold, providing a first response to the first subquery that is responsive to a first operation that receives the first subquery as input and providing a second response to the second subquery that is responsive to a second operation that receives the second subquery as input.

Subquery generation from a query

A method for dynamically generating training data for a model includes receiving a transcript corresponding to a conversation between a customer and an agent, the transcript comprising a customer input and an agent input. The method includes receiving a logic model including a plurality of responses, each response of the plurality of responses representing a potential reply to the customer input. The method further includes selecting, based on the agent input, a response from the plurality of responses of the logic model. The method includes determining that a similarity score between the selected response and the agent input satisfies a similarity threshold, and, based on determining that the similarity score between the selected response and the agent input satisfies the similarity threshold, training a machine learning model using the customer input and the selected response.

Building bots from raw logs and computing coverage of business logic graph

Aspects of the disclosure provide for a system for navigating a conversation graph using a language model trained to generate Application Programming Interface (API) calls in response to natural language input from a user computing device. A conversational agent implementing a state handler and a language model (LM) communicates with a user computing device through a user frontend. Rather than communicating directly with a user with output in natural language, the agent uses a (LM) trained as described herein to navigate a conversation graph. The state handler receives API calls generated by the LM and updates the state of a conversation with a user as indicated in the graph. After the update, the state handler can perform one or more predetermined actions associated with a node indicating the current state of the conversation.

Conversation graph navigation with language model

Implementations are described herein for analyzing existing graphical user interfaces ('GUIs') to facilitate automatic interaction with those GUIs, e.g., by automated assistants or via other user interfaces, with minimal effort from the hosts of those GUIs. For example, in various implementations, a user intent to interact with a particular GUI may be determined based at least in part on a free-form natural language input. Based on the user intent, a target visual cue to be located in the GUI may be identified, and object recognition processing may be performed on a screenshot of the GUI to determine a location of a detected instance of the target visual cue in the screenshot. Based on the location of the detected instance of the target visual cue, an interactive element of the GUI may be identified and automatically populate with data determined from the user intent.

Analyzing graphical user interfaces to facilitate automatic interaction

Implementations are set forth herein for creating an order of execution for actions that were requested by a user, via a spoken utterance to an automated assistant. The order of execution for the requested actions can be based on how each requested action can, or is predicted to, affect other requested actions. In some implementations, an order of execution for a series of actions can be determined based on an output of a machine learning model, such as a model that has been trained according to supervised learning. A particular order of execution can be selected to mitigate waste of processing, memory, and network resources—at least relative to other possible orders of execution. Using interaction data that characterizes past performances of automated assistants, certain orders of execution can be adapted over time, thereby allowing the automated assistant to learn from past interactions with one or more users.

Orchestrating execution of a series of actions requested to be performed via an automated assistant

Implementations are directed to determining, based on a submitted query that is a compound query, that a set of multiple sub-queries are collectively an appropriate interpretation of the compound query. Those implementations are further directed to providing, in response to such a determination, a corresponding command for each of the sub-queries of the determined set. Each of the commands is to a corresponding agent (of one or more agents), and causes the agent to generate and provide corresponding responsive content. Those implementations are further directed to causing content to be rendered in response to the submitted query, where the content is based on the corresponding responsive content received in response to the commands.

Determining responsive content for a compound query based on a set of generated sub-queries

Implementations set forth herein relate to an automated assistant that can provide a selectable action intent suggestion when a user is accessing a third party application that is controllable via the automated assistant. The action intent can be initialized by the user without explicitly invoking the automated assistant using, for example, an invocation phrase (e.g., 'Assistant . . . '). Rather, the user can initialize performance of the corresponding action by identifying one or more action parameters. In some implementations, the selectable suggestion can indicate that a microphone is active for the user to provide a spoken utterance that identifies a parameter(s). When the action intent is initialized in response to the spoken utterance from the user, the automated assistant can control the third party application according to the action intent and any identified parameter(s).

Automated assistant performance of a non-assistant application operation(s) in response to a user input that can be limited to a parameter(s)

Implementations set forth herein relate to an automated assistant that can interact with applications that may not have been pre-configured for interfacing with the automated assistant. The automated assistant can identify content of an application interface of the application to determine synonymous terms that a user may speak when commanding the automated assistant to perform certain tasks. Speech processing operations employed by the automated assistant can be biased towards these synonymous terms when the user is accessing an application interface of the application. In some implementations, the synonymous terms can be identified in a responsive language of the automated assistant when the content of the application interface is being rendered in a different language. This can allow the automated assistant to operate as an interface between the user and certain applications that may not be rendering content in a native language of the user.

Growing community of inventors

Zurich, Switzerland

Joseph Lange