Google Inc.

Ilya Firman

Adam Coimbra

Jasper S Lin

Ulas Kirazci

Carey Radebaugh

Yu-Kuan Lin

Jason Brant Douglas

Juliana Tsang

Mark D Womack

Gaurav Bhaya

Bradley M Abrams

Ficus Kirkpatrick

Christian Sonntag

Andrew Devron Stadler

Quazi Hussain

Sheng-chi Hsieh

Steve Chen

Keun Soo Yim

Nikhil Rao

Michael Siliski

Linda K Nguyen

Christoph Emanuel Studer

Luv Kothari

Anuj Thakkar

Aurash Jonathan Mahbod

Yan Huang

Jonathan Brunsman

Chaitrali Amrutkar

Ognjen Nikolic

Owen Lin

Anil Iyer

Gabriel Taubman

Sheng-Chi Hsieh

Marcin M Nowak-Przygodzki

Sunny Goyal

James Stout

Joseph Lange

Mugurel Ionut Andreica

Thomas Weedon Hume

Abhanshu Sharma

Svetoslav Ganov

Cliff Kuang

Mikhail Reutov

Gökhan Bakir

Jindong Chen

Michael Oh

Jacek Szmigiel

Robert Berry

Kyle Pedersen

Adam Cohen

Tilman Sporkert

Kalyana Ram Desineni

Saurabh Kumar

Mehdi Alizadeh

Sanjana Patel

Mubaraq Mishra

Vidhya Bhat

Reed Enger

Vishwath Mohan

Yunfan Jin

Gilles Baechler

Christoph Studer

Linda Nguyen

Mark Womack

The present disclosure is directed to automating user interactions in an application. In particular, the computing device can determine, based on the input data, an intent associated with performing an action on the computing device. Additionally, the computing device can invoke, using an application-level assistant component configured to provide an application-level interface on the computing device, execution of an application. Moreover, the computing device can transmit, to a system-level assistant component configured to provide a system-level interface with the computing device, instructions to interface with an actionable component of the application. The instructions can be transmitted using a trusted intermediary component configured to facilitate an interface between the system-level assistant component and the application-level assistant component. Furthermore, the computing device can engage, using the system-level assistant component, the actionable component of the application to perform the action.

Human-in-the-loop voice automation system

Consolidation of disparate data source queries is provided. An application intake component can receive an application package file, and process the file to identify one or more actions and entities for the application. A natural language processor component can receive an input audio signal to identify an action and an entity on which to perform the action. An application delivery component can identify applications that can perform the action and applications that have access to the entity, and then select an application that is configured to perform the action on the entity. The application delivery component can provide an indication of the application selected by the system to a client device.

Consolidation of responses from queries to disparate data sources

Systems and methods of invoking functions of agents via digital assistant applications are provided. Each action-inventory can have an address template for an action by an agent. The address template can include a portion having an input variable used to execute the action. A data processing system can parse an input audio signal from a client device to identify a request and a parameter to be executed by the agent. The data processing system can select an action-inventory for the action corresponding to the request. The data processing system can generate, using the address template, an address. The address can include a substring having the parameter used to control execution of the action. The data processing system can direct an action data structure including the address to the agent to cause the agent to execute the action and to provide output for presentation.

Invoking functions of agents via digital assistant applications using address templates

Indexing application actions for voice-based execution is provided. An indexer of a device receives, from an application executed by the device, an indication of an action and a deep link corresponding to the action declared by the application via an application programming interface. The indexer accesses an index stored in the memory of the device that is accessible to a plurality of applications. The indexer inputs, into a location in the index, the action and the deep link with a cryptographic token that prevents access to the deep link by unauthorized applications. A digital assistant of the device receives a voice query detected by a microphone and parses the voice query to determine the action. The digital assistant identifies in the index the deep link corresponding to the action. The digital assistant invokes, with the cryptographic token, the deep link to cause the application to perform the action.

Indexing application actions for voice-based execution

A method of encoding and decoding user interface interactions includes receiving a query from a user captured by an assistant-enabled device associated with the user. The query includes a user intent for interacting with an application. The method includes obtaining, for the application, a set of categorized actions. Each categorized action of the set of categorized actions is associated with one or more parameters and represents a high-level description of the user intent of the user interacting with the application. For each respective categorized action of the set of categorized actions, the method includes selecting a respective script associated with the respective categorized action that performs one or more low-level interactions with the application and executing the respective script to represent the user intent for interacting with the application.

Encoding/decoding user interface interactions

Analyzing or debugging applications is provided. The system identifies an action for an application provided by a developer. The system determines a first classification score based on historical execution of the action. The system generates a machine generated action for the application based on metadata associated with the application. The system determines a second classification score based on a comparison of the action with the machine generated action. The system selects, via a matching program, a second application that matches the application. The system determines a third classification score based on a comparison of an action approved for the second application with the action provided by the application developer. The system updates a delivery control parameter based on the first classification score, the second classification score and the third classification score. The system controls delivery of the application based on the delivery control parameter.

Debugging applications for delivery via an application delivery server

Implementations set forth herein relate to an automated assistant that can interact with applications that may not have been pre-configured for interfacing with the automated assistant. The automated assistant can identify content of an application interface of the application to determine synonymous terms that a user may speak when commanding the automated assistant to perform certain tasks. Speech processing operations employed by the automated assistant can be biased towards these synonymous terms when the user is accessing an application interface of the application. In some implementations, the synonymous terms can be identified in a responsive language of the automated assistant when the content of the application interface is being rendered in a different language. This can allow the automated assistant to operate as an interface between the user and certain applications that may not be rendering content in a native language of the user.

Automated assistant control of non-assistant applications via identification of synonymous term and/or speech processing biasing

Dynamic interfacing with applications is provided. For example, a system receives a first input audio signal. The system processes, via a natural language processing technique, the first input audio signal to identify an application. The system activates the application for execution on the client computing device. The application declares a function the application is configured to perform. The system modifies the natural language processing technique responsive to the function declared by the application. The system receives a second input audio signal. The system processes, via the modified natural language processing technique, the second input audio signal to detect one or more parameters. The system determines that the one or more parameters are compatible for input into an input field of the application. The system generates an action data structure for the application. The system inputs the action data structure into the application, which executes the action data structure.

Interfacing with applications via dynamically updating natural language processing

The present disclosure is generally directed to the generation of voice-activated data flows in interconnected network. The voice-activated data flows can include input audio signals that include a request and are detected at a client device. The client device can transmit the input audio signal to a data processing system, where the input audio signal can be parsed and passed to the data processing system of a service provider to fulfill the request in the input audio signal. The present solution is configured to conserve network resources by reducing the number of network transmissions needed to fulfill a request.

Activation of remote devices in a networked system

Growing community of inventors

Sunnyvale, CA, United States of America

Ilya Firman