Amazon Technologies, Inc.

Sankaranarayanan Ananthakrishnan

Rahul Gupta

Spyridon Matsoukas

Chengwei Su

Shirin Saleem

Kelly James Vanee

Premkumar Natarajan

Alborz Geramifard

Melanie Chie Bomke Gens

Anoop Kumar

Anil Ramakrishna

Chungnam Chan

Sriram Venkatapathy

Kavya Ravikumar

Mallory McManamon

Yugang Li

John Will Crimmins

John Pelak

Luca Soldaini

Techniques for using a federated learning framework to update machine learning models for spoken language understanding (SLU) system are described. The system determines which labeled data is needed to update the models based on the models generating an undesired response to an input. The system identifies users to solicit labeled data from, and sends a request to a user device to speak an input. The device generates labeled data using the spoken input, and updates the on-device models using the spoken input and the labeled data. The updated model data is provided to the system to enable the system to update the system-level (global) models.

Spoken language understanding models

Techniques for processing a user input are described. Text data representing a user input is processed with respect to at least one finite state transducer (FST) to generate at least one FST hypothesis. Context information may be required to traverse one or more paths of the at least one FST. The text data is also processed using at least one statistical model (e.g., perform intent classification, named entity recognition, and/or domain classification processing) to generate at least one statistical model hypothesis. The at least one FST hypothesis and the at least one statistical model hypothesis are input to a reranker that determines a most likely interpretation of the user input.

Natural language understanding processing

A natural language understanding system that can determine an overall score for a natural language hypothesis using hypothesis-specific component scores from different aspects of NLU processing as well as context data describing the context surrounding the utterance corresponding to the natural language hypotheses. The individual component scores may be input into a feature vector at a location corresponding to a type of a device captured by the utterance. Other locations in the feature vector corresponding to other device types may be populated with zero values. The feature vector may also be populated with other values represent other context data. The feature vector may then be multiplied by a weight vector comprising trained weights corresponding to the feature vector positions to determine a new overall score for each hypothesis, where the overall score incorporates the impact of the context data. Natural language hypotheses can be ranked using their respective new overall scores.

Contextual natural language processing

A natural language processing system that can determine an overall score for a natural language hypothesis using hypothesis-specific component scores from different aspects of NLU processing. The individual component scores may be weighted by weights trained to optimize the overall scores relative to each other. Each domain of the system may be configured with a separate component that determines the overall score with respect to the domain. Natural language hypotheses can be ranked using the overall score either within a specific domain or on a cross-domain basis.

Scoring of natural language processing hypotheses

A system that configures a device's operation based on the device's environment. The system may receive scene data describing a scene in which the device will operate. The scene data may include image data, audio data, or other data. A feature vector comprising the scene data may be processed to identify one or more categories to be associated with the scene. Various processing techniques, such as using Bayesian nonparametric models, may be used to categorize the scene data. The device may then adjust its operation based on the one or more selected categories.

Configuring system operation using image data

Growing community of inventors

Belmont, MA, United States of America

Sankaranarayanan Ananthakrishnan