At&t Intellectual Property I, L.p.

Microsoft Technology Licensing, LLC

Google Inc.

Amazon Technologies, Inc.

At+t Corp.

Nuance Communications, Inc.

Interactions Corporation

Other

University of Texas System

Dilek Zeynep Hakkani-Tur

Gokhan Tur

Giuseppe Riccardi

Larry Paul Heck

Mazin G Rahim

Bernard S Renger

Li Deng

Fethiye Asli Celikyilmaz

Ananlada Chotimongkol

Mazin E Gilbert

Jianfeng Gao

Allen Louis Gorin

Giuseppe Di Fabbrizio

Ruhi Sarikaya

Junlan Feng

Aleksandra Faust

Rukmini Iyer

Asli Celikyilmaz

Yun-Nung Chen

David Crawford Gibbon

Behzad Shahraray

Lee Begeja

Zhu Liu

Yang Liu

Jeremy Huntley Wright

Robert Elias Schapire

Abhinav Kumar Rastogi

Min Tang

Yucel Saygin

Guiseppe Riccardi

Pararth Shah

Malcolm G Slaney

Rohit Prasad

Ye-Yi Wang

Benoit F Dumoulin

Arindam Mandal

Andreas Stolcke

Elizabeth E Shriberg

Barbara B Hollister

Angeliki Metallinou

Tagyoung Chung

James M Wilson

Tirso M Alonso

Daniel Leon Stern

Joel Joseph Chengottusseriyil

Zhaleh Feizollahi

Govindarajan Sundaram Thattai

Feiyang Niu

Qing Ping

Marek Fiser

Qiaozi Gao

Murat Akbacak

Dustin Hillard

Juliana Kew

Jackson J Liscombe

Ilana Bromberg

Lawrence Lyon Rose

Di Jin

Anna Prokofieva

Izzeddin Gur

Ulrich Rueckert

Heeyoung Lee

Srinivas Bangalore

Brian Scott Amento

Jay Gordon Wilpon

Yun-Cheng Ju

Sarangarajan Parthasarathy

Lisa Joy Stifelman

Larry Stead

Xiaodong He

Raghav Gupta

Xiaohu Liu

Saurabh Gupta

Nikko Strom

Alicia Abella

Geoffrey G Zweig

Fileno A Alleva

Madhusudan Chinthakunta

Anna Chen Santos

Seokhwan Kim

Minwoo Jeong

Daniel Boies

Premkumar Natarajan

Raefer Christopher Gabriel

Anoop Deoras

Suranjit Adhikari

Behnam Hedayatnia

Sungjin Lee

Yun-Nung Vivian Chen

Yelin Kim

Thomas Irvine Nelson

Dong Han Wang

Joshua Seth Levy

Aishwarya Naresh Reganti

Giuseppe DiFabbrizio

Chien-wei Lin

Ye Yi Wang

Anish Acharya

Qing Liu

Ashley Fidler

A processing unit can train a model as a joint multi-domain recurrent neural network (JRNN), such as a bi-directional recurrent neural network (bRNN) and/or a recurrent neural network with long-short term memory (RNN-LSTM) for spoken language understanding (SLU). The processing unit can use the trained model to, e.g., jointly model slot filling, intent determination, and domain classification. The joint multi-domain model described herein can estimate a complete semantic frame per query, and the joint multi-domain model enables multi-task deep learning leveraging the data from multiple domains. The joint multi-domain recurrent neural (JRNN) can leverage semantic intents (such as, finding or identifying, e.g., a domain specific goal) and slots (such as, dates, times, locations, subjects, etc.) across multiple domains.

Multi-domain joint semantic frame parsing

Techniques for generating a natural language response to a user input of a dialog are described. A system receives a natural language user input of a dialog and determines dialog history data including a previous natural language user input of the dialog. Based on the first natural language user input and the dialog history data, the system generates at least a first question associated with the natural language user input. Based on the first natural language user input and the dialog history data, the system generates at least a first answer to the at least first question. Using the dialog history data, the first natural language question, and the first natural language answer, the system generates an output responsive to the natural language user input.

Natural language response generation

Implementations relate to using deep reinforcement learning to train a model that can be utilized, at each of a plurality of time steps, to determine a corresponding robotic action for completing a robotic task. Implementations additionally or alternatively relate to utilization of such a model in controlling a robot. The robotic action determined at a given time step utilizing such a model can be based on: current sensor data associated with the robot for the given time step, and free-form natural language input provided by a user. The free-form natural language input can direct the robot to accomplish a particular task, optionally with reference to one or more intermediary steps for accomplishing the particular task. For example, the free-form natural language input can direct the robot to navigate to a particular landmark, with reference to one or more intermediary landmarks to be encountered in navigating to the particular landmark.

Controlling a robot based on free-form natural language input

Techniques for generating opinion-based content responsive to a user input are described. The system may receive a user input, and determine dialog context data corresponding to a dialog between a user and the system, and including the user input. The system may determine generation of content responsive to the user input requires opinion-based knowledge, and may extract entities from the dialog context data, and determine natural language data of a knowledge base that includes entities similar to the extracted entities. The system may processes the natural language data and the dialog context data to determine a subset of the natural language data that is responsive to the user input. The system may generate output data responsive to the user input using the responsive natural language data and the dialog context.

Opinion-based natural language response generation

The present disclosure is generally directed to methods, apparatus, and computer-readable media (transitory and non-transitory) for learning to automatically navigate interactive web documents and/or websites. More particularly, various approaches are presented for training various deep Q network (DQN) agents to perform various tasks associated with reinforcement learning, including hierarchical reinforcement learning, in challenging web navigation environments with sparse rewards and large state and action spaces. These agents include a web navigation agent that can use learned value function(s) to automatically navigate through interactive web documents, as well as a training agent, referred to herein as a 'meta-trainer,' that can be trained to generate synthetic training examples. Some approaches described herein may be implemented when expert demonstrations are available. Other approaches described herein may be implemented when expert demonstrations are not available. In either case, dense, potential-based rewards may be used to augment the training.

Automatic navigation of interactive web documents

Methods, systems, and apparatus, including computer programs encoded on computer storage media for dialogue systems. A transcription of a user utterance is obtained. The transcription of the utterance is tokenized to identify multiple tokens for the utterance. Token-level utterance encodings corresponding to different tokens of the transcription are generated. A system action encoding from data indicating system actions previously performed by the dialogue system are generated. A dialogue context vector based on the utterance encoding and the system action encoding are generated. The token-level utterance encodings, the system action encoding, and the dialogue context vector are processed using a slot tagger to produce token-level output vectors. A limited set of candidate token classifications for the tokens of the user utterance are determined based on the token-level utterance encodings. A response for output is provided in response to the user utterance.

Language understanding and dialogue state tracking in dialogue systems

A device capable of autonomous motion may move in an environment and may receive audio data from a microphone. If the device receives a command represented in the audio data that is absent from a set of known commands, the device may prompt the user to explain how to perform the command. The device may save a command template corresponding to the command, which may be used to perform future commands.

Autonomously motile device with command processing

Devices and techniques are generally described for learning personalized concepts for natural language processing. In various examples, a first natural language input may be received. In some examples, a determination may be made that the first natural language input comprises non-actionable slot data. A dialog session may be initiated with the user. In some examples, first slot data that is indicated by the user during the dialog session may be determined. In various examples, data representing the first slot data may be stored in a database in association with the first natural language input.

Natural language processing

Determining a dialog state of an electronic dialog that includes an automated assistant and at least one user, and performing action(s) based on the determined dialog state. The dialog state can be represented as one or more slots and, for each of the slots, one or more candidate values for the slot and a corresponding score (e.g., a probability) for each of the candidate values. Candidate values for a slot can be determined based on language processing of user utterance(s) and/or system utterance(s) during the dialog. In generating scores for candidate value(s) of a given slot at a given turn of an electronic dialog, various features are determined based on processing of the user utterance and the system utterance using a memory network. The various generated features can be processed using a scoring model to generate scores for candidate value(s) of the given slot at the given turn.

Determining state of automated assistant dialog

A processing unit can train a model as a joint multi-domain recurrent neural network (JRNN), such as a bi-directional recurrent neural network (bRNN) and/or a recurrent neural network with long-short term memory (RNN-LSTM) for spoken language understanding (SLU). The processing unit can use the trained model to, e.g., jointly model slot filling, intent determination, and domain classification. The joint multi-domain model described herein can estimate a complete semantic frame per query, and the joint multi-domain model enables multi-task deep learning leveraging the data from multiple domains. The joint multi-domain recurrent neural network (JRNN) can leverage semantic intents (such as, finding or identifying, e.g., a domain specific goal) and slots (such as, dates, times, locations, subjects, etc.) across multiple domains.

Described herein is a system for improving sentiment detection and/or recognition using multiple inputs. For example, an autonomously motile device is configured to generate audio data and/or image data and perform sentiment detection processing. The device may process the audio data and the image data using a multimodal temporal attention model to generate sentiment data that estimates a sentiment score and/or a sentiment category. In some examples, the device may also process language data (e.g., lexical information) using the multimodal temporal attention model. The device can adjust its operations based on the sentiment data. For example, the device may improve an interaction with the user by estimating the user's current emotional state, or can change a position of the device and/or sensor(s) of the device relative to the user to improve an accuracy of the sentiment data.

Multimodal sentiment detection

A processing unit can extract salient semantics to model knowledge carryover, from one turn to the next, in multi-turn conversations. Architecture described herein can use the end-to-end memory networks to encode inputs, e.g., utterances, with intents and slots, which can be stored as embeddings in memory, and in decoding the architecture can exploit latent contextual information from memory, e.g., demographic context, visual context, semantic context, etc. e.g., via an attention model, to leverage previously stored semantics for semantic parsing, e.g., for joint intent prediction and slot tagging. In examples, architecture is configured to build an end-to-end memory network model for contextual, e.g., multi-turn, language understanding, to apply the end-to-end memory network model to multiple turns of conversational input; and to fill slots for output of contextual, e.g., multi-turn, language understanding of the conversational input. The neural network can be learned using backpropagation from output to input using gradient descent optimization.

End-to-end memory networks for contextual language understanding

A dialog generator receives data corresponding to desired dialog, such as application programming interface (API) information and sample dialog. A first model corresponding to an agent simulator and a second model corresponding to a user simulator take turns creating a plurality of dialog outlines of the desired dialog. The dialog generator may determine that one or more additional APIs are relevant to the dialog and may create further dialog outlines related thereto. The dialog outlines are converted to natural dialog to generate the dialog.

Goal-oriented dialog generation using dialog template, API, and entity data

Growing community of inventors

Denville, NJ, United States of America

Dilek Zeynep Hakkani-Tur