Amazon Technologies, Inc.

Qingming Tang

Chao Wang

Ming Sun

Chieh-Chi Kao

Viktor Rozgic

James G Droppo

Chia-Jung Chang

Qin Zhang

Sumit Garg

Rong Chen

Yang Liu

Spyridon Matsoukas

Xiuye Chen

Sripal Mehta

Andrew Thomas Bydlon

Steve Mark Lorusso

Sumig Garg

James Garnet Droppo

A system configured to detect custom acoustic events, where the system generates an acoustic event profile for the custom acoustic event based on a natural language description and without a sample of the sound. The system may generate a profile for a new custom sound based on a natural language description provided by the user; for example, a 'microwave beep.' If the system does not have an existing event profile for a microwave beep, the system may ask the user questions to determine whether any existing event profiles are close (e.g., is the sound similar to a “fan,” “alarm,” “appliance beep,” etc.). The system may detect an event that may be a possible match for the custom sound and ask the user to verify whether the detected event corresponds to the custom sound. The system may update the event profile based on the user's response.

Acoustic event detection

A system may include a first acoustic event detection (AED) component configured to detect a predetermined set of acoustic events, and include a second AED component configured to detect custom acoustic events that a user configures a device to detect. The first and second AED components are configured to perform task-specific processing, and may receive as input the same acoustic feature data corresponding to audio data that potentially represents occurrence of one or more events. Based on processing by the first and second AED components, a device may output data indicating that one or more acoustic events occurred, where the acoustic events may be a predetermined acoustic event and/or a custom acoustic event.

A system may be configured to detect custom acoustic events, where the system generates an acoustic event profile for the custom acoustic event based on a natural language description provided by a user and using an audio sample of the described acoustic event. For example, the user may describe the custom acoustic event as 'dog bark.' The system may ask the user questions to refine the description (e.g., dog breed, dog gender, age, etc.). Using an audio sample of the refined description, the system may then determine that audio captured in the user's environment is a potential sample of the custom acoustic event. Such captured audio may be presented to the user for confirmation, and then may be used to detect future occurrences of the custom acoustic event in the user's environment.

An acoustic event detection system may employ self-supervised federated learning to update encoder and/or classifier machine learning models. In an example operation, an encoder may be pre-trained to extract audio feature data from an audio signal. A decoder may be pre-trained to predict a subsequent portion of audio data (e.g., a subsequent frame of audio data represented by log filterbank energies). The encoder and decoder may be trained using self-supervised learning to improve the decoder's predictions and, by extension, the quality of the audio feature data generated by the encoder. The system may apply federated learning to share encoder updates across user devices. The system may fine-tune the classifier to improve inferences based on the improved audio feature data. The system may distribute classifier updates to the user device(s) to update the on-device classifier.

Self-supervised federated learning

An acoustic event detection system may employ one or more recurrent neural networks (RNNs) to extract features from audio data, and use the extracted features to determine the presence of an acoustic event. The system may use self-attention to emphasize features extracted from portions of audio data that may include features more useful for detecting acoustic events. The system may perform self-attention in an iterative manner to reduce the amount of memory used to store hidden states of the RNN while processing successive portions of the audio data. The system may process the portions of the audio data using the RNN to generate a hidden state for each portion. The system may calculate an interim embedding for each hidden state. An interim embedding calculated for the last hidden state may be normalized to determine a final embedding representing features extracted from the input data by the RNN.

Streaming self-attention in a neural network

Described herein is a system for media presence detection in audio. The system analyzes audio data to recognize whether a given audio segment contains sounds from a media source as a way of differentiating recorded media source sounds from other live sounds. In exemplary embodiments, the system includes a hierarchical model architecture for processing audio data segments, where individual audio data segments are processed by a trained machine learning model operating locally, and another trained machine learning model provides historical and contextual information to determine a score indicating the likelihood that the audio data segment contains sounds from a media source.

Growing community of inventors

Chicago, IL, United States of America

Qingming Tang