Microsoft Technology Licensing, LLC

Jayant Sivarama Krishnamurthy

Jesse Daniel Eskes Rusak

David Ernesto Heekin Burkett

Jonathan Reed Harris

Ashok Kuppusamy

Jaclyn Carley Knapp

Richard Leigh Mains

Michael Francis Palermiti, Ii

Chad Steven Estes

Harris Syed

Allison Anne O'Mahony

Jacob Daniel Andreas

Diego Hernan Carlomagno

Talon Edward Ireland

Jose Miguel Rady Allende

David Leo Wright Hall

John P Bufe, Iii

Jason Andrew Wolfe

Andrew Paul McGovern

Alan Xinyu Guo

Viktoriya Taranov

Daniel Louis Klein

Adam David Pauls

Joshua James Clausman

Yuchen Zhang

Dmitrij Petters

Andrei Vorobev

Tudor Buzasu Klein

Sergiy Gavrylenko

Alexander J Kolmykov-Zotov

Pengyu Chen

Zachary John Tellman

Andrew Paul Mcgovern

Various embodiments discussed herein enable client applications to be heavily integrated with a voice assistant in order to perform commands associated with voice utterances of users via voice assistant functionality and also seamlessly cause client applications to automatically perform native functions as part of executing the voice utterance. Such heavy integration also allows particular embodiments to support multi-modal input from a user for a single conversational interaction. In this way, client application user interface interactions, such as clicks, touch gestures, or text inputs are executed alternative or in addition to the voice utterances.

Voice assistant-enabled client application with user view context and multi-modal input support

Techniques for adapting previously-annotated training examples into updated training examples for training a machine learning model are disclosed. One example includes a computer program that recognizes a find expression, a replacement expression, and a filtering constraint in which the filtering constraint distinguishes a subset of previously-annotated training examples from others of the previously-annotated training examples. An instance of the find expression is identified by the computer program within the subset of the previously-annotated training examples that were identified among the previously-annotated training examples based on the filtering constraint. The instance of the find expression identified within the subset of the previously-annotated training examples is replaced by the computer program with an instance of the replacement expression to obtain an updated subset of training examples. The updated subset of training examples is output by the computer program, which may be used for training a machine learning model.

Updating training examples for artificial intelligence

Various embodiments discussed herein enable client applications to be heavily integrated with a voice assistant in order to both perform commands associated with voice utterances of users via voice assistant functionality and also seamlessly cause client applications to automatically perform native functions as part of executing the voice utterance. For example, some embodiments can automatically and intelligently cause a switch to a page the user needs and automatically and intelligently cause a population of particular fields of the page the user needs based on a user view context and the voice utterance.

Voice assistant-enabled client application with user view context

A computer-implemented method of responding to a conversational event is presented. The method comprises receiving a conversational event at a conversational computing interface. Based on the received conversational event, an applicable generation rule of a plurality of candidate generation rules is selected. The applicable generation rule is configured with one or more parameters. A computer-executable plan is then selected based on the selected generation rule. The one or more parameters are passed from the selected generation rule to one or more additional generation rules. The one or more additional generation rules configured with the one or more parameters are recursively applied to extend the selected computer-executable plan. One or more candidate responses to the conversational event are output via the conversational computing interface based on the recursive application of the one or more additional generation rules configured with the one or more parameters.

Response generation for conversational computing interface

A method of leveraging a dialogue history of a conversational computing interface to execute an updated dialogue plan. The method comprises maintaining an annotated dialogue history of the conversational computing interface. The annotated dialogue history includes a plurality of traced steps defining a data-flow including input data used to execute a context-dependent operation and output data recorded from a previous execution of the context-dependent operation. The method further comprises recognizing an updated dialogue plan including a prefix of executable steps and an updated executable step following the prefix. The method further comprises automatically computer-recognizing that the prefix of executable steps of the updated dialogue plan matches a corresponding prefix of traced steps in the annotated dialogue history. The method further comprises re-using the data-flow from the prefix of traced steps in the annotated dialogue history to automatically determine input data of the updated executable step.

Leveraging dialogue history in updated dialogue

A computer-implemented method of responding to a conversational event. The method comprises enacting, by a conversational computing interface, an initial computer-executable plan based on a conversational event received by the conversational computing interface, wherein the initial computer-executable plan is configured to output an initial value based on the conversational event. The method further comprises selecting, by the conversational computing interface, an extended computer-executable plan based on determining that the initial value is insufficient for generating an extended description responsive to the conversational event. The method further comprises enacting, by the conversational computing interface, the extended computer-executable plan to output additional information beyond what the initial computer-executable plan is configured to output, the additional information sufficient for generating the extended description responsive to the conversational event. The method further comprises outputting, via the conversational computing interface, a response to the conversational event including the extended description.

A method of configuring a conversational computing interface. The method comprises maintaining a branching dialogue representing a plurality of multi-turn conversations each including a plurality of turns, wherein a turn indicates one or both of a conversational computing interface action and a conversation event, the branching dialogue including a shared prefix of turns common to the plurality of multi-turn conversations and at least two different counterfactual branches descending from the shared prefix of turns, each different counterfactual branch including one or more descendent turns corresponding to a different multi-turn conversation of the plurality of multi-turn conversations. The method further comprises providing the branching dialogue in a computer-readable format traversable to retrieve a particular multi-turn conversation of the plurality of multi-turn conversations, the particular multi-turn conversation represented by the shared prefix of turns and a particular counterfactual branch including descendent turns corresponding to the particular multi-turn conversation.

Counterfactual annotated dialogues for conversational computing

A method comprising recognizing a user utterance including an ambiguity. The method further comprises using a previously-trained code-generation machine to produce, from the user utterance, a data-flow program including a search-history function. The search-history function is configured to select a highest-confidence disambiguating concept from one or more candidate concepts stored in a context-specific dialogue history.

Ambiguity resolution with dialogue search history

A method, comprising recognizing a user utterance for processing. The method further comprises using a previously-trained code-generation machine to generate, from the user utterance, a data-flow program configured to produce a return value upon successful execution. The method further comprises beginning execution of the data-flow program. Responsive to reaching an error condition resulting from execution of the data-flow program, the method further comprises, prior to the data-flow program producing the return value, suspending execution of the data flow program. The method further comprises using the previously-trained code-generation machine to generate an error-handling data-flow program, wherein the error-handling data-flow program is configured to produce the return value; beginning execution of the error-handling data-flow program to produce the return value; and outputting the return value.

Error recovery for conversational systems

Growing community of inventors

Berkeley, CA, United States of America

Jayant Sivarama Krishnamurthy