Google Inc.

Other

David Kliger Elson

Shir Judith Yehoshua

David P Whipp

David Eisenberg

Benjamin Ross

Raj Agarwal

Kevin Jose Chavez

Sudeep Gandhe

Christa Wimberley

Jeremy Hylton

James Lawry

Christian Kaiserlian

David Agraz

Phong Thanh Pham

Bryan M Kressler

Stanislav Plamenov Angelov

Joseph Robert Smarr

Hugo Santos

As part of a dialog session between a user and an automated assistant, implementations can process, using a streaming ASR model, a stream of audio data to generate ASR output, process, using an NLU model, the ASR output to generate NLU output, and generate, based on the NLU output, a stream of fulfillment data. Further, implementations can further determine, based on processing the stream of audio data, audio-based characteristics associated with spoken utterance(s) captured in the stream of audio data. Based on a current state of the stream of NLU output, the stream of fulfillment data, and the audio-based characteristics, implementations can determine whether a next interaction state to be implemented is: (i) causing fulfillment output to be implemented; (ii) causing natural conversation output to be audibly rendered; or (iii) refrain from causing any interaction to be implemented, can cause the next interaction state to be implemented.

Enabling natural conversations for an automated assistant

Systems and methods provides for dialog management in real-time rather than turn taking. An example method included generating first candidate responses to triggering event. The triggering event may be receipt of a live stream chunk for the dialog or receipt of a backend response to a previous backend request for a dialog shema. The method also includes updating a list of candidate responses that are accepted or pending with at least on of the first candidate responses, and determining, for the triggering event, whether the list of candidate responses includes a candidate response that has a confidence score that meets a triggering threshold. The method also includes waiting for a next triggering event without providing a candidate response when the list does not include a candidate response that has a confidence score that meets the triggering threshold.

Streaming real-time dialog management

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for managing dialogs. In one aspect, a method includes receiving a request associated with a task from a user device; submitting the request to each of a plurality of distinct data providers; receiving a plurality of suggested dialog responses from two or more of the data providers; scoring the one or more suggested dialog responses based on one or more scoring factors; determining a particular dialog response to provide to the user based on the scoring; and providing the determined dialog response to the user device.

Managing dialog data providers

Systems and methods provides for dialog management in real-time rather than turn taking. An example method included generating first candidate responses to triggering event. The triggering event may be receipt of a live stream chunk for the dialog or receipt of a backend response to a previous backend request for a dialog schema. The method also includes updating a list of candidate responses that are accepted or pending with at least on of the first candidate responses, and determining, for the triggering event, whether the list of candidate responses includes a candidate response that has a confidence score that meets a triggering threshold. The method also includes waiting for a next triggering event without providing a candidate response when the list does not include a candidate response that has a confidence score that meets the triggering threshold.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using frames for performing tasks. One of the methods includes receiving a first request to perform a task, the first request comprising user speech identifying the task; generating a frame associated with the task, wherein the frame comprises one or more types of values necessary to perform the task, and wherein each type of value can be satisfied by a respective value; receiving a second request to provide information related to a question, the second request comprising user speech identifying the question; providing information identifying the question to a search engine, and receiving a response identifying one or more terms; determining that at least one term can satisfy a type of value necessary to perform the task; and storing the at least one term in the frame.

Using frames for action dialogs

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for managing dialogs. In one aspect, a method includes receiving a request to perform a task from a user device; obtaining a dialog corresponding to the task; providing multiple protocol buffers to the user device; receiving a voice input and one or more annotated protocol buffers from the user device, the one or more annotated protocol buffers identifying corresponding non-verbal responses to content in the protocol buffers; and using the received protocol buffers to update a state of the dialog and to interpret the voice input.

Multimodal state circulation

Systems and methods for annotating conversations between a user and an electronic assistant in a hierarchical manner are disclosed. An example system includes a data store of hierarchical annotation records. The hierarchical annotation records include root records and non-root records. Each root record has a conversation identifier and an annotation identifier. Each non-root record includes an annotation identifier and a parent identifier, the parent identifier being an annotation identifier of another record. At least some of the non-root records further include an annotation tag. The system generates a first new root record and first non-root records for an input dialog act and generates a second new root record and second non-root records for an output dialog act that is responsive to the input dialog act. At least one of the second non-root records includes an annotation tag linking back to a record of the first non-root records.

Hierarchical annotation of dialog acts

Systems and methods provide for dialog management in real-time rather than turn taking. An example method includes generating first candidate responses to a triggering event. The triggering event may be receipt of a live-stream chunk for the dialog or receipt of a backend response to a previous backend request for a dialog schema. The method also includes updating a list of candidate responses that are accepted or pending with at least one of the first candidate responses, and determining, for the triggering event, whether the list of candidate responses includes a candidate response that has a confidence score that meets a triggering threshold. The method also includes waiting for a next triggering event without providing a candidate response when the list does not include a candidate response that has a confidence score that meets the triggering threshold.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for ranking search results. One of the methods includes identifying one or more sessions for a query and associating watch times of the respective resources watched in the sessions with the query. One or more watch time signals are calculated for a first resource and the query based on the watch times associated with the query. A first search result responsive to the query is obtained, wherein the first search result identifies the first resource and has an associated score S. A new score S' is calculated based on a least S and a watch time function, the watch time function being a function of the one or more watch time signals. The new score S&#x2032; is provided to a process for ranking search results including the first search result.

Growing community of inventors

Brooklyn, NY, United States of America

David Kliger Elson