Microsoft Technology Licensing, LLC

Google Inc.

Yahoo! Inc.

Nuance Communications, Inc.

Samsung Electronics Co., Ltd.

Sri International

Larry Paul Heck

Dilek Zeynep Hakkani-Tur

Lisa Joy Stifelman

Gokhan Tur

Madhusudan Chinthakunta

David Mitby

Ruhi Sarikaya

Anne K Sullivan

Tasos Anastasakos

Fethiye Asli Celikyilmaz

Daniel Boies

Adam Elman

Sarangarajan Parthasarathy

Shane Jeremy Landry

Rukmini Iyer

Alejandro Acero

Li Deng

Jianfeng Gao

Malcolm G Slaney

Xiaodong He

Benoit F Dumoulin

Kamran Rajabi Zargahi

Andreas Stolcke

Elizabeth E Shriberg

Abhinav Kumar Rastogi

Ken H Thai

Dilek Zeynep Hakkani-Tür

Murat Akbacak

Po-Sen Huang

Heeyoung Lee

Stephanos Tryphonas

Anna Prokofieva

Hongxia Jin

Oriol Vinyals

Sungsoo Kim

Brian Strope

Ye-Yi Wang

Xiaohu Liu

Belle L Tseng

Julian James Odell

Kwangyoun Kim

Daniel R Dole

Chanwoo Kim

Thomas L Dean

Minwoo Jeong

Michael H Cohen

Ankur Bapna

Howard Scott Roy

Mitchel Weintraub

Minkyu Shin

Zhaleh Feizollahi

Dhananjaya N Gowda

Yangfeng Ji

Abhinav Garg

Pararth Shah

Dong Han Wang

Chung Kei Wong

Shalini Ghosh

Keke Chen

Gordon Guo-Zheng Sun

James M Riseman

Jennifer E Balogh

N Nikki Mirghafori

Mehul Kumar

Peter M Bretan

Ali Mamdouh Elkahky

Ashley Fidler

Fehtiye Asli Celikyilmaz

Rongqing Lu

Naghmeh N Mirghafori

The disclosure relates to an electronic apparatus for recognizing user voice and a method of recognizing, by the electronic apparatus, the user voice. According to an embodiment, the method of recognizing the user voice includes obtaining an audio signal segmented into a plurality of frame units, determining an energy component for each filter bank by applying a filter bank distributed according to a preset scale to a frequency spectrum of the audio signal segmented into the frame units, smoothing the determined energy component for each filter bank, extracting a feature vector of the audio signal based on the smoothed energy component for each filter bank, and recognizing the user voice in the audio signal by inputting the extracted feature vector to a voice recognition model.

Voice recognition device and method

Determining a dialog state of an electronic dialog that includes an automated assistant and at least one user, and performing action(s) based on the determined dialog state. The dialog state can be represented as one or more slots and, for each of the slots, one or more candidate values for the slot and a corresponding score (e.g., a probability) for each of the candidate values. Candidate values for a slot can be determined based on language processing of user utterance(s) and/or system utterance(s) during the dialog. In generating scores for candidate value(s) of a given slot at a given turn of an electronic dialog, various features are determined based on processing of the user utterance and the system utterance using a memory network. The various generated features can be processed using a scoring model to generate scores for candidate value(s) of the given slot at the given turn.

Determining state of automated assistant dialog

Determining slot value(s) based on received natural language input and based on descriptor(s) for the slot(s). In some implementations, natural language input is received as part of human-to-automated assistant dialog. A natural language input embedding is generated based on token(s) of the natural language input. Further, descriptor embedding(s) are generated (or received), where each of the descriptor embeddings is generated based on descriptor(s) for a corresponding slot that is assigned to a domain indicated by the dialog. The natural language input embedding and the descriptor embedding(s) are applied to layer(s) of a neural network model to determine, for each of the slot(s), which token(s) of the natural language input correspond to the slot. A command is generated that includes slot value(s) for slot(s), where the slot value(s) for one or more of slot(s) are determined based on the token(s) determined to correspond to the slot(s).

Processing natural language using machine learning to determine slot values based on slot descriptors

A method includes receiving a non-linguistic input associated with an input musical content. The method also includes, using a model that embeds multiple musical features describing different musical content and relationships between the different musical content in a latent space, identifying one or more embeddings based on the input musical content. The method further includes at least one of: (i) identifying stored musical content based on the one or more identified embeddings or (ii) generating derived musical content based on the one or more identified embeddings. In addition, the method includes presenting at least one of: the stored musical content or the derived musical content. The model is generated by training a machine learning system having one or more first neural network components and one or more second neural network components such that embeddings of the musical features in the latent space have a predefined distribution.

Techniques for learning effective musical features for generative and retrieval-based applications

Improving accuracy in understanding and/or resolving references to visual elements in a visual context associated with a computerized conversational system is described. Techniques described herein leverage gaze input with gestures and/or speech input to improve spoken language understanding in computerized conversational systems. Leveraging gaze input and speech input improves spoken language understanding in conversational systems by improving the accuracy by which the system can resolve references&#x2014;or interpret a user's intent&#x2014;with respect to visual elements in a visual context. In at least one example, the techniques herein describe tracking gaze to generate gaze input, recognizing speech input, and extracting gaze features and lexical features from the user input. Based at least in part on the gaze features and lexical features, user utterances directed to visual elements in a visual context can be resolved.

Eye gaze for spoken language understanding in multi-modal conversational interactions

Natural language query translation may be provided. A statistical model may be trained to detect domains according to a plurality of query click log data. Upon receiving a natural language query, the statistical model may be used to translate the natural language query into an action. The action may then be performed and at least one result associated with performing the action may be provided.

Translating natural language utterances to keyword search queries

An augmented conversational understanding architecture may be provided. Upon receiving a natural language phrase from a user, the phrase may be translated into a search phrase and a search action may be performed on the search phrase.

Augmented conversational understanding architecture

Identification of user intents may be provided. A plurality of network applications may be identified, and an ontology associated with each of the plurality of applications may be defined. If a phrase received from a user is associated with at least one of the defined ontologies, an action associated with the network application may be executed.

Task driven user intents

Tools are provided to allow developers to enable applications for Conversational Understanding (CU) using assets from a CU service. The tools may be used to select functionality from existing domains, extend the coverage of one or more domains, as well as to create new domains in the CU service. A developer may provide example Natural Language (NL) sentences that are analyzed by the tools to assist the developer in labeling data that is used to update the models in the CU service. For example, the tools may assist a developer in identifying domains, determining intent actions, determining intent objects and determining slots from example NL sentences. After the developer tags all or a portion of the example NL sentences, the models in the CU service are automatically updated and validated. For example, validation tools may be used to determine an accuracy of the model against test data.

Building conversational understanding systems using a toolset

Prosodic features are used for discriminating computer-directed speech from human-directed speech. Statistics and models describing energy/intensity patterns over time, speech/pause distributions, pitch patterns, vocal effort features, and speech segment duration patterns may be used for prosodic modeling. The prosodic features for at least a portion of an utterance are monitored over a period of time to determine a shape associated with the utterance. A score may be determined to assist in classifying the current utterance as human directed or computer directed without relying on knowledge of preceding utterances or utterances following the current utterance. Outside data may be used for training lexical addressee detection systems for the H-H-C scenario. H-C training data can be obtained from a single-user H-C collection and that H-H speech can be modeled using general conversational speech. H-C and H-H language models may also be adapted using interpolation with small amounts of matched H-H-C data.

Prosodic and lexical addressee detection

Techniques are described herein that are capable of suggesting intent frame(s) for user request(s). For instance, the intent frame(s) may be suggested to elicit a request from a user. An intent frame is a natural language phrase (e.g., a sentence) that includes at least one carrier phrase and at least one slot. A slot in an intent frame is a placeholder that is identified as being replaceable by one or more words that identify an entity and/or an action to indicate an intent of the user. A carrier phrase in an intent frame includes one or more words that suggest a type of entity and/or action that is to be identified by the one or more words that may replace the corresponding slot. In accordance with these techniques, the intent frame(s) are suggested in response to determining that natural language functionality of a processing system is activated.

Suggesting intent frame(s) for user request(s)

Techniques are described related to turn-based reinforcement learning for dialog management. In various implementations, dialog states and corresponding responsive actions generated during a multi-turn human-to-computer dialog session may be obtained. A plurality of turn-level training instances may be generated, each including: a given dialog state of the plurality of dialog states at an outset of a given turn of the human-to-computer dialog session; and a given responsive action that was selected based on the given dialog state. One or more of the turn-level training instances may further include a turn-level feedback value that reflects on the given responsive action selected during the given turn. A reward value may be generated based on an outcome of the human-to-computer dialog session. The dialog management policy model may be trained based on turn-level feedback values of the turn-level training instance(s) and the reward value.

Turn-based reinforcement learning for dialog management

Individuals may utilize devices to engage in conversations about topics respectively associated with a location (e.g., restaurants where the individuals may meet for dinner). Often, the individual momentarily withdraws from the conversation in order to issue commands to the device to retrieve and present such information, and may miss parts of the conversation while interacting with the device. Additionally, the individual often explores such topics individually on a device and conveys such information to the other individuals through messages, which is inefficient and error-prone. Presented herein are techniques enabling devices to facilitate conversations by monitoring the conversation for references, by one individual to another (rather than as a command to the device), to a topic associated with a location. In the absence of a command from an individual, the device may automatically present a map alongside a conversation interface showing the location(s) of the topic(s) referenced in the conversation.

Facilitating conversations with automated location mapping

Growing community of inventors

Los Altos, CA, United States of America

Larry Paul Heck