At&t Intellectual Property I, L.p.

Microsoft Technology Licensing, LLC

Nuance Communications, Inc.

Jason D Williams

Geoffrey G Zweig

Ethan Selfridge

Barbara B Hollister

Tirso M Alonso

Ilya Dan Melamed

David Maxwell Chickering

Carlos Garcia Jurado Suarez

Suhrid Balakrishnan

Aparna Lakshmiratan

Umar Syed

Nobal Bikram Niraula

Andrey Kolobov

Pradeep Dasigi

Bongshin Lee

Saleema Amin Amershi

Lihong Li

Donghao Ren

Jina Suh

Systems, methods, and computer-readable storage devices for crowd-sourced data labeling. The system requests a respective response from each of a set of entities. The set of entities includes crowd workers. Next, the system incrementally receives a number of responses from the set of entities until one of an accuracy threshold is reached and m responses are received, wherein the accuracy threshold is based on characteristics of the number of responses. Finally, the system generates an output response based on the number of responses.

System and method for crowd-sourced data labeling

Domain-specific language understanding models that may be built, tested and improved quickly and efficiently are provided. Methods, systems and devices are provided that enable a developer to build user intent detection models, language entity extraction models, and language entity resolution models quickly and without specialized machine learning knowledge. These models may be built and implemented via single model systems that enable the models to be built in isolation or in an end-to-end pipeline system that enables the models to be built and improved in a simultaneous manner.

Generation of language understanding systems and methods

Methods, computer systems, computer-storage media, and graphical user interfaces are provided for improving performance of a multi-class classifier. An interactive graphical user interface includes an item representation display area that displays a plurality of item representations corresponding to a plurality of items processed by a multi-class classifier. The classifier's performance can be visualized using bidirectional bar graphs displaying true positives, false positives, and false negatives for each class.

Interactive performance visualization of multi-class classifier

Disclosed herein are systems, methods, and non-transitory computer-readable storage media for advanced turn-taking in an interactive spoken dialog system. A system configured according to this disclosure can incrementally process speech prior to completion of the speech utterance, and can communicate partial speech recognition results upon finding particular conditions. A first condition which, if found, allows the system to communicate partial speech recognition results, is that the most recent word found in the partial results is statistically likely to be the termination of the utterance, also known as a terminal node. A second condition is the determination that all search paths within a speech lattice converge to a common node, also known as a pinch node, before branching out again. Upon finding either condition, the system can communicate the partial speech recognition results. Stability and correctness probabilities can also determine which partial results are communicated.

System and method for advanced turn-taking for interactive spoken dialog systems

A dialog state tracking system. One aspect of the system is the use of multiple utterance decoders and/or multiple spoken language understanding (SLU) engines generating competing results that improve the likelihood that the correct dialog state is available to the system and provide additional features for scoring dialog state hypotheses. An additional aspect is training a SLU engine and a dialog state scorer/ranker DSR engine using different subsets from a single annotated training data set. A further aspect is training multiple SLU/DSR engine pairs from inverted subsets of the annotated training data set. Another aspect is web-style dialog state ranking based on dialog state features using discriminative models with automatically generated feature conjunctions. Yet another aspect is using multiple parameter sets with each ranking engine and averaging the rankings. Each aspect independently improves dialog state tracking accuracy and may be combined in various combinations for greater improvement.

Dialog state tracking using web-style ranking and multiple language understanding engines

Disclosed herein are systems, computer-implemented methods, and tangible computer-readable storage media for generating a natural language spoken dialog system. The method includes nominating a set of allowed dialog actions and a set of contextual features at each turn in a dialog, and selecting an optimal action from the set of nominated allowed dialog actions using a machine learning algorithm. The method includes generating a response based on the selected optimal action at each turn in the dialog. The set of manually nominated allowed dialog actions can incorporate a set of business rules. Prompt wordings in the generated natural language spoken dialog system can be tailored to a current context while following the set of business rules. A compression label can represent at least one of the manually nominated allowed dialog actions.

System and method for generating manually designed and automatically optimized spoken dialog systems

Disclosed herein are systems, methods, and computer-readable storage media for estimating reliability of alternate speech recognition hypotheses. A system configured to practice the method receives an N-best list of speech recognition hypotheses and features describing the N-best list, determines a first probability of correctness for each hypothesis in the N-best list based on the received features, determines a second probability that the N-best list does not contain a correct hypothesis, and uses the first probability and the second probability in a spoken dialog. The features can describe properties of at least one of a lattice, a word confusion network, and a garbage model. In one aspect, the N-best lists are not reordered according to reranking scores. The determination of the first probability of correctness can include a first stage of training a probabilistic model and a second stage of distributing mass over items in a tail of the N-best list.

System and method for estimating the reliability of alternate speech recognition hypotheses in real time

Disclosed herein are systems, computer-implemented methods, and computer-readable storage media for generating personalized user models. The method includes receiving automatic speech recognition (ASR) output of speech interactions with a user, receiving an ASR transcription error model characterizing how ASR transcription errors are made, generating guesses of a true transcription and a user model via an expectation maximization (EM) algorithm based on the error model and the respective ASR output where the guesses will converge to a personalized user model which maximizes the likelihood of the ASR output. The ASR output can be unlabeled. The method can include casting speech interactions as a dynamic Bayesian network with four variables: (s), (u), (r), (m), and encoding relationships between (s), (u), (r), (m) as conditional probability tables. At each dialog turn (r) and (m) are known and (s) and (u) are hidden.

System and method for generating user models from transcribed dialogs

Disclosed herein are systems, computer-implemented methods, and tangible computer-readable media for using alternate recognition hypotheses to improve whole-dialog understanding accuracy. The method includes receiving an utterance as part of a user dialog, generating an N-best list of recognition hypotheses for the user dialog turn, selecting an underlying user intention based on a belief distribution across the generated N-best list and at least one contextually similar N-best list, and responding to the user based on the selected underlying user intention. Selecting an intention can further be based on confidence scores associated with recognition hypotheses in the generated N-best lists, and also on the probability of a user's action given their underlying intention. A belief or cumulative confidence score can be assigned to each inferred user intention.

Growing community of inventors

Seattle, WA, United States of America

Jason D Williams