At&t Intellectual Property I, LP

Microsoft Technology Licensing, LLC

Sri International

Amazon Technologies, Inc.

Nuance Communications, Inc.

At+t Corp.

Other

Gokhan Tur

Dilek Zeynep Hakkani-Tur

Mazin G Rahim

Larry Paul Heck

Giuseppe Riccardi

William Scott Mark

Horacio E Franco

Michael John Wolverton

Bart Peintner

Neil Yorke-Smith

Bernard S Renger

Li Deng

Ananlada Chotimongkol

Mazin E Gilbert

Jianfeng Gao

Junlan Feng

Giuseppe Di Fabbrizio

Narendra Kumar Gupta

Necip Fazil Ayan

Girish Acharya

Fethiye Asli Celikyilmaz

Rukmini Iyer

Regis Vincent

Wen Wang

Asli Celikyilmaz

Andreas Kathol

Osher Yadgar

Venkatarama Satyanarayana Parimi

David Crawford Gibbon

Behzad Shahraray

Lee Begeja

Zhu Liu

Jeremy Huntley Wright

Robert Elias Schapire

Norman D Winarsky

Joel Joseph Chengottusseriyil

Min Tang

Antony Van Der Mude

Govindarajan Sundaram Thattai

Dilek Zeynep Hakkani-Tür

Qing Ping

Feiyang Niu

Yun-Nung Chen

Qiaozi Gao

Yucel Saygin

Robinson Piramuthu

Ruhi Sarikaya

Allen Louis Gorin

Ye-Yi Wang

Benoit F Dumoulin

Andreas Stolcke

Gregory Kent Myers

Elizabeth E Shriberg

Harry Bratt

Barbara B Hollister

Martin Graciarena

James M Wilson

Tirso M Alonso

Daniel Leon Stern

Luciana Ferrer

Murat Akbacak

Ilana Bromberg

Lawrence Lyon Rose

Dustin Hillard

Brian Scott Amento

Rohit Prasad

Jay Gordon Wilpon

Sarangarajan Parthasarathy

Yun-Cheng Ju

Yang Liu

Larry Stead

Xiaohu Liu

Alicia Abella

Chenlei Guo

Geoffrey G Zweig

Fileno A Alleva

Xing Fan

Seokhwan Kim

Shubham Katiyar

Minwoo Jeong

Daniel Boies

Premkumar Natarajan

Narendra Gyanchandani

Anoop Deoras

Xinhong Zhang

Yangfeng Ji

Eshan Bhatnagar

Yun-Nung Vivian Chen

Dong Han Wang

Sanqiang Zhao

Giuseppe DiFabbrizio

Puyang Xu

Aishwarya Naresh Reganti

Ye Yi Wang

Ashley Fidler

Fehtiye Asli Celikyilmaz

Xiaodeng He

Svetlana Stenchikova

Dilek Hakkani Tur

Divyanshu Brijmohan Verma

A processing unit can train a model as a joint multi-domain recurrent neural network (JRNN), such as a bi-directional recurrent neural network (bRNN) and/or a recurrent neural network with long-short term memory (RNN-LSTM) for spoken language understanding (SLU). The processing unit can use the trained model to, e.g., jointly model slot filling, intent determination, and domain classification. The joint multi-domain model described herein can estimate a complete semantic frame per query, and the joint multi-domain model enables multi-task deep learning leveraging the data from multiple domains. The joint multi-domain recurrent neural (JRNN) can leverage semantic intents (such as, finding or identifying, e.g., a domain specific goal) and slots (such as, dates, times, locations, subjects, etc.) across multiple domains.

Multi-domain joint semantic frame parsing

Techniques for processing with respect to a user input as contextual information is available are described. A system generates a first task prediction using first context data that is available when a user input is received. The system generates a second task prediction (e.g., updated first task prediction) when second context data is received, and then further generates a third task prediction when third context data is received. Example first context data may include device type information, time information, location, etc. Example second context data may include automatic speech recognition (ASR) data. Example third context data may include natural language understanding (NLU) data. Using the third task prediction, the system generates an output responsive to the user input.

Natural language processing system

Techniques for rendering visual content, in response to one or more utterances, are described. A device receives one or more utterances that define a parameter(s) for desired output content. A system (or the device) identifies natural language data corresponding to the desired content, and uses natural language generation processes to update the natural language data based on the parameter(s). The system (or the device) then generates an image based on the updated natural language data. The system (or the device) also generates video data of an avatar. The device displays the image and the avatar, and synchronizes movements of the avatar with output of synthesized speech of the updated natural language data. The device may also display subtitles of the updated natural language data, and cause a word of the subtitles to be emphasized when synthesized speech of the word is being output.

Virtual conversational companion

Described herein is a system for tracking objects and performing dynamic entity resolution using image data. For example, the system may build an environment map and populate the map with objects present in the environment. As the devices move about the environment it may capture image data and, based on its position and/or configuration of its components, may determine updated locations of objects that move in the environment. Upon receiving a query from a user, based on the location of the objects relative to the device/user, the system can interpret gestures and voice commands to infer which object is specified by the voice command. To build the environment map, the system performs object detection to generate bounding boxes associated with an object, then clusters the bounding boxes into a three-dimensional (3D) object associated with 3D coordinates. As the system tracks the object using the 3D coordinates while maintaining two-dimensional (2D) information (e.g., bounding boxes and other features), the system can use existing 2D models to process objects in 3D.

Object tracking and entity resolution

Devices and techniques are generally described for learning personalized responses to declarative natural language inputs. In various examples, a first natural language input may be received. The first natural language input may correspond to intent data corresponding to a declarative user input. In some examples, a dialog session may be initiated with the first user. An action intended by the first user for the first natural language input may be determined based at least in part on the dialog session. In various examples, first data representing the action may be stored in association with second data representing a state described by at least a portion of the first natural language input.

Natural language processing of declarative statements

A device capable of autonomous motion may move in an environment and may receive audio data from a microphone. If the device receives a command represented in the audio data that is absent from a set of known commands, the device may prompt the user to explain how to perform the command. The device may save a command template corresponding to the command, which may be used to perform future commands.

Autonomously motile device with command processing

Devices and techniques are generally described for learning personalized concepts for natural language processing. In various examples, a first natural language input may be received. In some examples, a determination may be made that the first natural language input comprises non-actionable slot data. A dialog session may be initiated with the user. In some examples, first slot data that is indicated by the user during the dialog session may be determined. In various examples, data representing the first slot data may be stored in a database in association with the first natural language input.

Natural language processing

A processing unit can train a model as a joint multi-domain recurrent neural network (JRNN), such as a bi-directional recurrent neural network (bRNN) and/or a recurrent neural network with long-short term memory (RNN-LSTM) for spoken language understanding (SLU). The processing unit can use the trained model to, e.g., jointly model slot filling, intent determination, and domain classification. The joint multi-domain model described herein can estimate a complete semantic frame per query, and the joint multi-domain model enables multi-task deep learning leveraging the data from multiple domains. The joint multi-domain recurrent neural network (JRNN) can leverage semantic intents (such as, finding or identifying, e.g., a domain specific goal) and slots (such as, dates, times, locations, subjects, etc.) across multiple domains.

A processing unit can extract salient semantics to model knowledge carryover, from one turn to the next, in multi-turn conversations. Architecture described herein can use the end-to-end memory networks to encode inputs, e.g., utterances, with intents and slots, which can be stored as embeddings in memory, and in decoding the architecture can exploit latent contextual information from memory, e.g., demographic context, visual context, semantic context, etc. e.g., via an attention model, to leverage previously stored semantics for semantic parsing, e.g., for joint intent prediction and slot tagging. In examples, architecture is configured to build an end-to-end memory network model for contextual, e.g., multi-turn, language understanding, to apply the end-to-end memory network model to multiple turns of conversational input; and to fill slots for output of contextual, e.g., multi-turn, language understanding of the conversational input. The neural network can be learned using backpropagation from output to input using gradient descent optimization.

End-to-end memory networks for contextual language understanding

Natural language query translation may be provided. A statistical model may be trained to detect domains according to a plurality of query click log data. Upon receiving a natural language query, the statistical model may be used to translate the natural language query into an action. The action may then be performed and at least one result associated with performing the action may be provided.

Translating natural language utterances to keyword search queries

Systems and methods for determining knowledge-guided information for a recurrent neural networks (RNN) to guide the RNN in semantic tagging of an input phrase are presented. A knowledge encoding module of a Knowledge-Guided Structural Attention Process (K-SAP) receives an input phrase and, in conjunction with additional sub-components or cooperative components generates a knowledge-guided vector that is provided with the input phrase to the RNN for linguistic semantic tagging. Generating the knowledge-guided vector comprises at least parsing the input phrase and generating a corresponding hierarchical linguistic structure comprising one or more discrete sub-structures. The sub-structures may be encoded into vectors along with attention weighting identifying those sub-structures that have greater importance in determining the semantic meaning of the input phrase.

Knowledge-guided structural attention processing

A method for assisting a user with one or more desired tasks is disclosed. For example, an executable, generic language understanding module and an executable, generic task reasoning module are provided for execution in the computer processing system. A set of run-time specifications is provided to the generic language understanding module and the generic task reasoning module, comprising one or more models specific to a domain. A language input is then received from a user, an intention of the user is determined with respect to one or more desired tasks, and the user is assisted with the one or more desired tasks, in accordance with the intention of the user.

Generic virtual personal assistant platform

The present invention relates to a method and apparatus for exploiting human feedback in an intelligent automated assistant. One embodiment of a method for conducting an interaction with a human user includes inferring an intent from data entered by the human user, formulating a response in accordance with the intent, receiving feedback from a human advisor in response to at least one of the inferring and the formulating, wherein the human advisor is a person other than the human user, and adapting at least one model used in at least one of the inferring and the formulating, wherein the adapting is based on the feedback.

Method and apparatus for exploiting human feedback in an intelligent automated assistant

Structured web pages are accessed and parsed to obtain implicit annotation for natural language understanding tasks. Search queries that hit these structured web pages are automatically mined for information that is used to semantically annotate the queries. The automatically annotated queries may be used for automatically building statistical unsupervised slot filling models without using a semantic annotation guideline. For example, tags that are located on a structured web page that are associated with the search query may be used to annotate the query. The mined search queries may be filtered to create a set of queries that is in a form of a natural language query and/or remove queries that are difficult to parse. A natural language model may be trained using the resulting mined queries. Some queries may be set aside for testing and the model may be adapted using in-domain sentences that are not annotated. The models may be tested using these implicitly annotated natural-language-like queries in an unsupervised fashion.

Exploiting structured content for unsupervised natural language semantic parsing

Growing community of inventors

Los Altos, CA, United States of America

Gokhan Tur