Google Inc.

David P Whipp

David Kliger Elson

Shir Judith Yehoshua

Sang Soo Sung

Lantian Zheng

Haywai Hayward Chan

Jing Qian

Chen Liu

Liuyi Sun

The disclosed embodiments include computerized methods, systems, and devices, including computer programs encoded on a computer storage medium, for integrating voice-based interaction and control into a native graphical user interface (GUI) of an executed application. For example, a communications device may obtaining component data identifying a plurality of components of a voice-user interface from a computing system maintained by a voice-service provider, and may execute an application linked to a corresponding one of the components of the voice-user interface. The communications device may generate the native GUI based on an output of the executed application, and may generate an interface element representative of the corresponding one of the components of the voice-user interface. The communications device may present the generated interface element within the native GUI, which may embed the corresponding component of the voice-user interface into the native GUI.

Component libraries for voice interaction services

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for managing dialogs. In one aspect, a method includes receiving a request associated with a task from a user device; submitting the request to each of a plurality of distinct data providers; receiving a plurality of suggested dialog responses from two or more of the data providers; scoring the one or more suggested dialog responses based on one or more scoring factors; determining a particular dialog response to provide to the user based on the scoring; and providing the determined dialog response to the user device.

Managing dialog data providers

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using frames for performing tasks. One of the methods includes receiving a first request to perform a task, the first request comprising user speech identifying the task; generating a frame associated with the task, wherein the frame comprises one or more types of values necessary to perform the task, and wherein each type of value can be satisfied by a respective value; receiving a second request to provide information related to a question, the second request comprising user speech identifying the question; providing information identifying the question to a search engine, and receiving a response identifying one or more terms; determining that at least one term can satisfy a type of value necessary to perform the task; and storing the at least one term in the frame.

Using frames for action dialogs

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for managing dialogs. In one aspect, a method includes receiving a request to perform a task from a user device; obtaining a dialog corresponding to the task; providing multiple protocol buffers to the user device; receiving a voice input and one or more annotated protocol buffers from the user device, the one or more annotated protocol buffers identifying corresponding non-verbal responses to content in the protocol buffers; and using the received protocol buffers to update a state of the dialog and to interpret the voice input.

Multimodal state circulation

Methods, systems, and apparatus, including computer programs stored on a computer-readable storage medium, for asynchronous execution of client requests. In some implementations, data indicating a user request to a digital assistant is received. An action corresponding to the user request is determined. It is determined that the action is classified as an action to be performed asynchronously to the user request. A confirmation message is sent, for output, and the action is performed asynchronously to the user request.

Growing community of inventors

San Jose, CA, United States of America

David P Whipp