Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Chao Wang

Ming Sun

Viktor Rozgic

Spyridon Matsoukas

Chieh-Chi Kao

Shiv Naga Prasad Vitaladevuni

Qingming Tang

Neil Cueto

Nicolas Alexander Bauer

Yixin Gao

Joon Shin

James G Droppo

Arindam Mandal

Shiva Kumar Sundaram

Anish N Shah

Daniel Kenneth Bone

Chia-Jung Chang

Qin Zhang

John Snavely

MacBeth Watson

Megan McMillan

Mathias Jourdain

Sumit Garg

Thibaud Senechal

Joonkyung Shin

Cengiz Erbas

Rong Chen

Chungnam Chan

Xiuye Chen

Aaron David Wirshba

Nipul Bharani

Jialiang Bao

Sheetal Laad

Harshavardhan Sundar

Gustavo Alfonso Aguilar Alas

Weiran Wang

David Earl Heckerman

David E Gardner

William Hong Vong

Yang Liu

Elizabeth E Shriberg

Siddhartha Reddy Jonnalagadda

Gengshen Fu

Huichao Xue

Timothy David Gasser

Eric Hsia

Xiaoqing Wang

Katherine M Ryan

Alexander Jonathan Pinkus

Alexandros Potamianos

Varun Kumar Nagaraja

Avani Parakh

Taylor Angaran Johnson

Cesare Ferri

Vamsi Kodamasimham

Pritesh Sharma

Stephen Mark Lorusso

Venkata Naga Krishna Chaitanya Puvvada

Jeremie Blehm

Thibaud Senechai

Srinivas Parthasarathy

Heran Zhang

Mohammad Taha Bahadori

Sripal Mehta

Andrew Thomas Bydlon

Steve Mark Lorusso

Li Jiang

Sumig Garg

James Garnet Droppo

The present application provides techniques for implementing a skill component, configured to perform an assessment of a user, as part of a speech processing system. The system may receive a natural language user input requesting assistance. The skill component may, using one or more machine learning models, determine at least one characteristic of the natural language input (e.g., lexical embedding, acoustic embedding, topic, tone, etc.). The skill component may determine state data for a present session, where the state data indicates a topic of the natural language user input and/or a user state associated with the natural language user input. The skill component may determine past state data of one or more past sessions, and generate a question to the user based on the state data for the natural language user input and the past state data.

Conversation-based skill component for assessing a user's state

A system may be configured to receive audio data input and determine whether the audio data corresponds to an acoustic event defined by an acoustic event profile. The system may be further configured to detect an ongoing acoustic event corresponding to multiple occurrences of an acoustic event type and, in response, execute a set of actions. The execution of the routine may be based on conditions corresponding to the ongoing acoustic event. The system may be configured to evaluate the occurrences based on the conditions of the ongoing acoustic event including whether the occurrences occur within a timer period, the relative time between occurrences, and whether at least one occurrence is detected within different portions of the time period. The acoustic event profile may be associated with a user profile. The system may execute a routine that sends a notification or directs a device to perform an action.

Acoustic event detection

A system configured to detect custom acoustic events, where the system generates an acoustic event profile for the custom acoustic event based on a natural language description and without a sample of the sound. The system may generate a profile for a new custom sound based on a natural language description provided by the user; for example, a 'microwave beep.' If the system does not have an existing event profile for a microwave beep, the system may ask the user questions to determine whether any existing event profiles are close (e.g., is the sound similar to a “fan,” “alarm,” “appliance beep,” etc.). The system may detect an event that may be a possible match for the custom sound and ask the user to verify whether the detected event corresponds to the custom sound. The system may update the event profile based on the user's response.

A system may include a first acoustic event detection (AED) component configured to detect a predetermined set of acoustic events, and include a second AED component configured to detect custom acoustic events that a user configures a device to detect. The first and second AED components are configured to perform task-specific processing, and may receive as input the same acoustic feature data corresponding to audio data that potentially represents occurrence of one or more events. Based on processing by the first and second AED components, a device may output data indicating that one or more acoustic events occurred, where the acoustic events may be a predetermined acoustic event and/or a custom acoustic event.

In an example embodiment, machine learning is used to train a machine-learned model that projects each entity, title pair into a single number, called a seniority score, to represent the career progression needed for that position. For example, company A's 'software engineer' and company B's “senior software engineer” can be represented as two separate numbers, one being p (company A, software engineer) and the other being p (company B, senior software engineer) on the same axis. This allows a comparison to be made about the absolute levels of each title despite their potential different meanings at different entities.

Machine learning to infer title levels across entities

A system may be configured to detect custom acoustic events, where the system generates an acoustic event profile for the custom acoustic event based on a natural language description provided by a user and using an audio sample of the described acoustic event. For example, the user may describe the custom acoustic event as 'dog bark.' The system may ask the user questions to refine the description (e.g., dog breed, dog gender, age, etc.). Using an audio sample of the refined description, the system may then determine that audio captured in the user's environment is a potential sample of the custom acoustic event. Such captured audio may be presented to the user for confirmation, and then may be used to detect future occurrences of the custom acoustic event in the user's environment.

Techniques for detecting certain acoustic events from audio data are described. A system may perform event aggregation for certain types of events before sending an output to a device representing the event is detected. The system may bypass the event aggregation process for certain types of events that the system may detect with a high level of confidence. In such cases, the system may send an output to the device when the event is detected. The system may be used to detect acoustic events representing presence of a person or other harmful circumstances (such as, fire, smoke, etc.) in a home, an office, a store, or other types of indoor settings.

An acoustic event detection system may employ self-supervised federated learning to update encoder and/or classifier machine learning models. In an example operation, an encoder may be pre-trained to extract audio feature data from an audio signal. A decoder may be pre-trained to predict a subsequent portion of audio data (e.g., a subsequent frame of audio data represented by log filterbank energies). The encoder and decoder may be trained using self-supervised learning to improve the decoder's predictions and, by extension, the quality of the audio feature data generated by the encoder. The system may apply federated learning to share encoder updates across user devices. The system may fine-tune the classifier to improve inferences based on the improved audio feature data. The system may distribute classifier updates to the user device(s) to update the on-device classifier.

Self-supervised federated learning

An acoustic event detection system may employ one or more recurrent neural networks (RNNs) to extract features from audio data, and use the extracted features to determine the presence of an acoustic event. The system may use self-attention to emphasize features extracted from portions of audio data that may include features more useful for detecting acoustic events. The system may perform self-attention in an iterative manner to reduce the amount of memory used to store hidden states of the RNN while processing successive portions of the audio data. The system may process the portions of the audio data using the RNN to generate a hidden state for each portion. The system may calculate an interim embedding for each hidden state. An interim embedding calculated for the last hidden state may be normalized to determine a final embedding representing features extracted from the input data by the RNN.

Streaming self-attention in a neural network

Described is a system and method that determines character sequences from speech, without determining the words of the speech, and processes the character sequences to determine sentiment data indicative of emotional state of a user that output the speech. The emotional state may then be presented or provided as an output to the user.

Character-level emotion detection

Described herein is a system for sentiment detection in audio data. The system processes audio frame level features of input audio data using a machine learning algorithm to classify the input audio data into a particular sentiment category. The machine learning algorithm may be a neural network trained using an encoder-decoder method. The training of the machine learning algorithm may include normalization techniques to avoid potential bias in the training data that may occur when the training data is annotated for a perceived sentiment of the speaker.

Sentiment detection in audio data

Described herein is a system for sentiment detection in audio data. The system is trained using acoustic information and lexical information to determine a sentiment corresponding to an utterance. In some cases when lexical information is not available, the system (trained on acoustic and lexical information) is configured to determine a sentiment using only acoustic information.

Multiple classifications of audio data

A system processes audio data to detect when it includes a representation of a wakeword or of an acoustic event. The system may receive or determine acoustic features for the audio data, such as log-filterbank energy (LFBE). The acoustic features may be used by a first, wakeword-detection model to detect the wakeword; the output of this model may be further processed using a softmax function, to smooth it, and to detect spikes. The same acoustic features may be also be used by a second, acoustic-event-detection model to detect the acoustic event; the output of this model may be further processed using a sigmoid function and a classifier. Another model may be used to extract additional features from the LFBE data; these additional features may be used by the other models.

Wakeword and acoustic event detection

Growing community of inventors

Newton, MA, United States of America

Chao Wang