Amazon Technologies, Inc.

Rahul Goel

Angeliki Metallinou

Spyridon Matsoukas

Melanie Chie Bomke Gens

Kellen Gillespie

Alexandra R Shapiro

Vishal Ishwar

Tagyoung Chung

Raefer Christopher Gabriel

Chandra Prakash Khatri

Anushree Venkatesh

Inderjit Dhillon

Behnam Hedayatnia

Anish Acharya

Sanghyun Yi

Multi-modal natural language processing systems are provided. Some systems are context-aware systems that use multi-modal data to improve the accuracy of natural language understanding as it is applied to spoken language input. Machine learning architectures are provided that jointly model spoken language input ('utterances') and information displayed on a visual display ('on-screen information'). Such machine learning architectures can improve upon, and solve problems inherent in, existing spoken language understanding systems that operate in multi-modal contexts.

Natural language processing with contextual data representing displayed content

Methods and systems for determining an intent of an utterance using contextual information associated with a requesting device are described herein. Voice activated electronic devices may, in some embodiments, be capable of displaying content using a display screen. Entity data representing the content rendered by the display screen may describe entities having similar attributes as an identified intent from natural language understanding processing. Natural language understanding processing may attempt to resolve one or more declared slots for a particular intent and may generate an initial list of intent hypotheses ranked to indicate which are most likely to correspond to the utterance. The entity data may be compared with the declared slots for the intent hypotheses, and the list of intent hypothesis may be re-ranked to account for matching slots from the contextual metadata. The top ranked intent hypothesis after re-ranking may then be selected as the utterance's intent.

Intent re-ranker

A system that can engage in a dialog with a user may select a system response to a user input based on how the system estimates a user may respond to a potential system response. Models may be trained to evaluate a potential system response in view of various available data including dialog history, entity data, etc. Each model may score the potential system response for various qualitative aspects such as whether the response is likely to be comprehensible, on-topic, interesting, likely to lead to the dialog continuing, etc. Such scores may be combined to other scores such as whether the potential response is coherent or engaging. The models may be trained using previous dialog/chatbot evaluation data. At runtime the scores may be used to select a system response to a user input as part of the dialog.

Dialog response generation

A natural language understanding (NLU) system that uses a reduced dimensionality of word embedding features to configure compressed NLU models that use reduced computing resources for NLU tasks. A modified NLU model may include a compressed vocabulary data structure of word embedding data vectors that include a set of values corresponding to a reduced dimensionality of the original word embedding features, resulting in a smaller sized vocabulary data structure and reduced size of the vocabulary data structure. Further components of the modified NLU model perform matrix operations to expand the dimensionality of the reduced word embedding data vectors up to the expected dimensionality of later layers of the NLU model. Additional training and reweighting can adjust for potential loses in performance resulting from reductions in the word embedding features. Thus the modified NLU model can achieve similar performance to an original NLU model with reductions in use of computing resources.

Natural language processing

Multi-modal natural language processing

Growing community of inventors

Mountain View, CA, United States of America

Rahul Goel