Microsoft Technology Licensing, LLC

Semantic Machines, Inc.

Jesse Daniel Eskes Rusak

David Leo Wright Hall

Daniel Louis Klein

Jayant Sivarama Krishnamurthy

Adam David Pauls

David Ernesto Heekin Burkett

Jacob Daniel Andreas

Daniel Lawrence Roth

Jordan Rian Cohen

Jonathan Reed Harris

Laurence Steven Gillick

Ashok Kuppusamy

Jaclyn Carley Knapp

Percy Shuo Liang

Richard Leigh Mains

Michael Francis Palermiti, Ii

Chad Steven Estes

Allison Anne O'Mahony

Harris Syed

Pengyu Chen

Diego Hernan Carlomagno

Talon Edward Ireland

Jose Miguel Rady Allende

Alan Xinyu Guo

John P Bufe, Iii

Jason Andrew Wolfe

Andrew Paul McGovern

Michael Jack Newman

Viktoriya Taranov

Andrew Volpe

Steven Andrew Wegmann

Yuchen Zhang

Andrei Vorobev

Taylor Darwin Berg-Kirkpatrick

Tudor Buzasu Klein

Sergiy Gavrylenko

Alexander J Kolmykov-Zotov

Andrew Paul Mcgovern

Jordan R Cohen

Various embodiments discussed herein enable client applications to be heavily integrated with a voice assistant in order to perform commands associated with voice utterances of users via voice assistant functionality and also seamlessly cause client applications to automatically perform native functions as part of executing the voice utterance. Such heavy integration also allows particular embodiments to support multi-modal input from a user for a single conversational interaction. In this way, client application user interface interactions, such as clicks, touch gestures, or text inputs are executed alternative or in addition to the voice utterances.

Voice assistant-enabled client application with user view context and multi-modal input support

Various embodiments discussed herein enable client applications to be heavily integrated with a voice assistant in order to both perform commands associated with voice utterances of users via voice assistant functionality and also seamlessly cause client applications to automatically perform native functions as part of executing the voice utterance. For example, some embodiments can automatically and intelligently cause a switch to a page the user needs and automatically and intelligently cause a population of particular fields of the page the user needs based on a user view context and the voice utterance.

Voice assistant-enabled client application with user view context

A method of training a conversational computing system. The method includes providing an initial query to a machine learning model previously trained to generate actions for responding to a query. Based on the initial query, a plurality of candidate actions are received from the machine learning model and presented via a training interface. Based on input provided by the human annotator, a selected candidate action is recognized and associated with the initial query in a labeled data example.

Training model with model-provided candidate action

A computer-implemented method of responding to a conversational event is presented. The method comprises receiving a conversational event at a conversational computing interface. Based on the received conversational event, an applicable generation rule of a plurality of candidate generation rules is selected. The applicable generation rule is configured with one or more parameters. A computer-executable plan is then selected based on the selected generation rule. The one or more parameters are passed from the selected generation rule to one or more additional generation rules. The one or more additional generation rules configured with the one or more parameters are recursively applied to extend the selected computer-executable plan. One or more candidate responses to the conversational event are output via the conversational computing interface based on the recursive application of the one or more additional generation rules configured with the one or more parameters.

Response generation for conversational computing interface

An automated natural dialogue system provides a combination of structure and flexibility to allow for ease of annotation of dialogues as well as learning and expanding the capabilities of the dialogue system based on natural language interactions.

Robust expandable dialogue system

A computer-implemented method of responding to a conversational event. The method comprises enacting, by a conversational computing interface, an initial computer-executable plan based on a conversational event received by the conversational computing interface, wherein the initial computer-executable plan is configured to output an initial value based on the conversational event. The method further comprises selecting, by the conversational computing interface, an extended computer-executable plan based on determining that the initial value is insufficient for generating an extended description responsive to the conversational event. The method further comprises enacting, by the conversational computing interface, the extended computer-executable plan to output additional information beyond what the initial computer-executable plan is configured to output, the additional information sufficient for generating the extended description responsive to the conversational event. The method further comprises outputting, via the conversational computing interface, a response to the conversational event including the extended description.

A method comprising recognizing a user utterance including an ambiguity. The method further comprises using a previously-trained code-generation machine to produce, from the user utterance, a data-flow program including a search-history function. The search-history function is configured to select a highest-confidence disambiguating concept from one or more candidate concepts stored in a context-specific dialogue history.

Ambiguity resolution with dialogue search history

A method for configuring an automated dialogue system uses traces of interactions via a graphical user interface (GUI) for an application. Each trace includes interactions in the context of a plurality of presentations of the GUI. Elements of one or more presentations of the GUI are identified, and templates are associated with portions of the trace. Each template has one or more defined inputs and a defined output. For each template of the plurality of templates, the portions of the traces are processed to automatically configure the template by specifying a procedure for providing values of inputs to the template via the GUI and obtaining a value of an output. The automated dialogue system is configured with the configured templates, thereby avoiding manual configuration of the dialogue system.

Automatic spoken dialogue script discovery

A method for generating training data for training a natural language processing system comprises loading, into a computer memory, a computer-readable transcript representing an ordered sequence of one or more dialogue events. The method further comprises acquiring a computer-readable command describing an exemplary ordered subsequence of one or more dialogue events from the computer-readable transcript. The method further comprises re-parametrizing the computer-readable command with an alternative semantic parameter. The method further comprises generating an alternative ordered subsequence of one or more dialogue events based on the re-parametrized computer-readable command. The method further comprises outputting, to a data store, an alternative computer-readable transcript including the alternative ordered subsequence of one or more dialogue events, the alternative computer-readable transcript having a predetermined format usable to train the computerized assistant.

Training natural language system with generated dialogues

A method, comprising recognizing a user utterance for processing. The method further comprises using a previously-trained code-generation machine to generate, from the user utterance, a data-flow program configured to produce a return value upon successful execution. The method further comprises beginning execution of the data-flow program. Responsive to reaching an error condition resulting from execution of the data-flow program, the method further comprises, prior to the data-flow program producing the return value, suspending execution of the data flow program. The method further comprises using the previously-trained code-generation machine to generate an error-handling data-flow program, wherein the error-handling data-flow program is configured to produce the return value; beginning execution of the error-handling data-flow program to produce the return value; and outputting the return value.

Error recovery for conversational systems

A computing device includes a display configured to present a graphical user interface. The graphical user interface includes a transcript portion configured to display an unannotated transcript representing an ordered sequence of one or more dialogue events involving a client and a computerized assistant, at least one of the dialogue events taking the form of an example client utterance, and an annotation portion configured to display a hierarchical menu including a plurality of candidate utterance annotations. An utterance annotation machine is configured to receive one or more computer inputs selecting, for each of one or more response parameters in the example client utterance, utterance annotations from the hierarchical menu that collectively define a machine-readable interpretation of the example client utterance. An annotated utterance having a predetermined format usable to train the computerized assistant is output to a data store based on the example client utterance.

Utterance annotation user interface

A method for a dialogue system includes establishing a dialogue session between an application executing on a server and a remote machine. The dialogue session includes one or more utterances received from a user at the remote machine. A natural language processing machine identifies a request associated with a computer-readable representation of an utterance. A dialogue expansion machine generates a plurality of alternative actions for responding to the request. A previously-trained machine learning confidence model assesses a confidence score for each alternative. If a highest confidence score for a top alternative does not satisfy a threshold, the plurality of alternatives including the top alternative are transmitted to a remote machine (which may be the same remote machine or a different remote machine) for review by a human reviewer. After the dialogue system and/or the human reviewer select an alternative, computer-readable instructions defining the selected alternative are executed.

Rapid deployment of dialogue system

Growing community of inventors

Somerville, MA, United States of America

Jesse Daniel Eskes Rusak