Microsoft Technology Licensing, LLC

Amazon Technologies, Inc.

Microsfot Technology Licensing, LLC

Ruhi Sarikaya

Young-Bum Kim

Alexandre Rochette

Paul Anthony Crook

Omar Zia Khan

Daniel Boies

Larry Paul Heck

Xiaohu Liu

Jean-Philippe Robichaud

Anjishnu Kumar

Tasos Anastasakos

Marius Alexandru Marin

Arpit Gupta

Rohit Prasad

Fethiye Asli Celikyilmaz

Vasiliy Radostev

Zhaleh Feizollahi

Dilek Zeynep Hakkani-Tur

Xing Fan

Joe Pemberton

Kerry Hammil

Abhinav Sethy

Anoop Deoras

Chetan Nagaraj Naik

Spyridon Matsoukas

Nikko Strom

Stephen Frederick Potter

David Robert Thomas

Vipul Agarwal

Frédéric Johan Georges Deramat

Minwoo Jeong

Shubham Katiyar

Rajesh Kumar Pandey

Isaac Joseph Madwed

Julia Kennedy Nemer

Mohamed Farouk AbdelHady

Puyang Xu

Arun Kumar Thenappan

Nikhil Holenarsipur Ramesh

Jihwan Lee

Gokhan Tur

Lambert Leo Mathias

Asli Celikyilmaz

Ryan Scott Aldrich

Diamond Bishop

Savas Parastatidis

Bala Murali Krishna Ummaneni

Dean Curtis

Pushpendre Rastogi

Nitin Ashok Jain

John Arland Nave

Mayank Kumar

Prasha Shrestha

Nakul Dahiwade

Arshdeep Singh

Nikhil Reddy Kortha

Hao Yuan

Rekha Seshadrinathan

Hung Tuan Pham

Dong Yu

Li Deng

Kai Chieh Liu

Xiaodong He

Chenlei Guo

Geoffrey G Zweig

Fileno A Alleva

Mei-Yuh Hwang

Hisami Suzuki

Kaisheng N Yao

Hung The Pham

Shiladitya Roy

Ravi Kumar Bikkula

Simon Peter Reavely

Alborz Geramifard

Divya Jetley

Huinan Ren

Dongchan Kim

Bigyan Rajbhandari

Dustin Hillard

Zhenyu Yao

Zheng Du

Danko Panic

Ravikiran Janardhana

Arima Vu Ram Thayumanavar

Zheng Ma

Wangyao Ge

Bradford Jason Snow

Jerrin Thomas Elanjikal

Sriharsha Venkata Chintalapati

Gregoire Mesnil

Ali Mamdouh Elkahky

Nikhil Ramesh

Paul Crook

A system for processing user utterances and/or text based queries that tracks entities and other context data of a current dialog between the system and the user and can fill slots for new intents of the dialog by performing statistical processing on previously mentioned entities with respect to current slots to be filled. The system may compare a previously mentioned entity to a current slot to be filled using vector representations, such as word embeddings, of the current utterance, dialog history, current intent, name of an entity under consideration, category of the current slot to be filled, distance between the current dialog turn and the dialog turn that mentioned the entity, and other considerations. The individual vectors may be weighted according to an attention operation and processed by a trained decoder to output a score indicating whether the entity in consideration is relevant to the particular slot. In this manner, slots may be filled using entities from previous dialog turns, thus performing statistical anaphora resolution and leading to improved system performance.

Speech processing dialog management

Techniques for determining a command or intent likely to be subsequently invoked by a user of a system are described. A user inputs a command (either via a spoken utterance or textual input) to a system. The system determines content responsive to the command. The system also determines a second command or corresponding intent likely to be invoked by the user subsequent to the previous command. Such determination may involve analyzing pairs of intents, with each pair being associated with a probability that one intent of the pair will be invoked by a user subsequent to a second intent of the pair. The system then outputs first content responsive to the first command and second content soliciting the user as to whether the system to execute the second command.

Proactive command framework

Techniques for generating a visual response to a user input are described. A system may receive a natural language input and use a machine learning model to determine a first component is to determine a response to the natural language input while a second component is to determine supplemental content related to the natural language input. The system may receive, from the first component, first image data corresponding to the response. The system may also receive, from the second component, second image data corresponding to the supplemental content. The system may send, to a display, a command to present the first image data and the second image data.

Visual responses to user inputs

Techniques for recommending a skill experience to a user after a user-system dialog session has ended are described. Upon a dialog session ending, the system uses a first machine learning model to determine potential intents to recommend to a user. The system then uses a second machine learning model to determine a particular skill and intent to recommend. The system then prompts the user to accept the recommended skill and intent. If the user accepts, the system calls the recommended skill to execute. As part of calling the skill, the system sends to the skill at least one entity provided in a natural language user input of the ended dialog session. This enables the skill to skip welcome prompts, and initiate processing to output a response based on the intent and the at least one entity of the ended dialog session.

User-system dialog expansion

Techniques for performing centralized unsuperivised learning in a multi-domain system are described. A user may request labeled data for an ML task, where the request includes a prompt for obtaining relevant explicit user feedback. The system may use the prompt to collect explicit user feedback for relevant runtime user inputs. After a duration of time (in the user's request for labeled data) has elapsed, the system determines whether collected user feedback indicates processing of the user input was defective and, if so, determines a cause of the defective processing. The system then uses one or more label generators to generate labeled data using the collected user feedback, whether the processing was defective, and the determined defect cause.

Machine learning label generation

In non-limiting examples of the present disclosure, systems, methods and devices for providing personalized experiences to a computing device based on user input such as voice, text and gesture input are provided. Acoustic patterns associated with voice input, speech patterns, language patterns and natural language processing may be used to identify a specific user providing input from a plurality of users, identify user background characteristics and traits for the specific user, and topically categorize user input in a tiered hierarchical index. Topically categorized user input may be supplemented with user data and world knowledge and personalized responses and feedback for an identified specific user may be provided reactively and proactively.

Personalization of experiences with digital assistants in communal settings through voice and query processing

Techniques for generating a visual response to a user input are described. A system may receive input data corresponding to a user input, determining a first skill component is to determine a response to the user input, and determine a second skill component is to determine supplemental content related to the user input. The system may also determine a template for presenting a visual response to the user input, where the template is configured for presenting the response and the supplemental content. The system may receive, from the first skill component, first image data corresponding to the first response. The system may also receive, from the second skill component, second image data corresponding to the first supplemental content. The system may send, to a device including a display, a command to present the first image data and the second image data using the template.

Conversational understanding systems allow users to conversationally interface with a computing device. In examples, a query may be received that includes a request for execution of a task. A data exchange task definition may be accessed. The data exchange task definition assists a conversational understanding system in managing task state tracking for information needed for task execution. Using the data exchange task definition, a per-turn policy for interacting with the user computing device is generated based on the state of a dialogue with a computing device and an evaluation of a process flow chart provided by a task owner resource. The task owner resource may be independent from the conversational understanding system. A response to the query may be generated and output based on the per-turn policy. In examples, the per-turn policy is used to generate one or more responses during a dialogue with a user via a computing device.

Policy authoring for task state tracking during dialogue

Devices and techniques are generally described for application determination in speech processing. Input data corresponding to a spoken utterance may be received. Speech recognition processing may be performed on the input data to generate text data. A machine learning encoder may generate a vector representation of the input data. A first binary classifier may determine a first probability that the input data corresponds to a first speech-processing application. A second binary classifier may determine a second probability that the input data corresponds to a second speech-processing application. A selection between the first speech-processing application and the second speech-processing application may be made based at least in part on the first probability and the second probability.

Skill shortlister for natural language processing

Devices and techniques are generally described for a speech processing routing architecture. In various examples, first data comprising a first feature definition is received. The first feature definition may include a first indication of first source data and first instructions for generating feature data using the first source data. In various examples, the feature data may be generated according to the first feature definition. In some examples, a speech processing system may receive a first request to process a first utterance. The feature data may be retrieved from a non-transitory computer-readable memory. The speech processing system may determine a first skill for processing the first utterance based at least in part on the feature data.

Natural language processing routing

Methods and systems are provided for discriminating ambiguous expressions to enhance user experience. For example, a natural language expression may be received by a speech recognition component. The natural language expression may include at least one of words, terms, and phrases of text. A dialog hypothesis set from the natural language expression may be created by using contextual information. In some cases, the dialog hypothesis set has at least two dialog hypotheses. A plurality of dialog responses may be generated for the dialog hypothesis set. The dialog hypothesis set may be ranked based on an analysis of the plurality of the dialog responses. An action may be performed based on ranking the dialog hypothesis set.

Discriminating ambiguous expressions to enhance user experience

Devices and techniques are generally described for using user feedback to determine routing decisions in a speech processing system. In various examples, first data representing a first utterance may be received. Second data representing a first semantic interpretation of the first utterance may be determined. A first intent data processing application may be selected for processing the second data. Feedback data may be determined related to the first intent data processing application processing the second data. Third data representing a semantic interpretation of a second utterance may be received, wherein the first semantic interpretation is the same as the second semantic interpretation. A second intent data processing application may be determined for processing the third data based at least in part on the feedback data.

Natural language processing

Routing of natural language inputs to speech processing applications

Growing community of inventors

Redmond, WA, United States of America

Ruhi Sarikaya