Amazon Technologies, Inc.

Aaron Challenner

Nikko Strom

Angeliki Metallinou

Gengshen Fu

Michael Thomas Peterson

Vincent Auvray

Ilya Sokolov

Ariya Rastrow

Arindam Mandal

Zoe Adams

Eli Joshua Fidler

Ying Shi

Siddhartha Reddy Jonnalagadda

Prakash Krishnan

Sumit Garg

Shuang Wu

Ahmed Abdelal

Anushree Venkatesh

Ameya Agaskar

Rong Chen

Veerdhawal Pande

Alexandros Potamianos

Sai Kiran Venkata Subramanya Rupanagudi

Nagaraj Mahajan

David Chi-Wai Tang

Minmin Shen

Arijit Biswas

Sree Hari Krishnan Parthasarathi

Terrence Yu

David McGuire

Siddharth Agrawal

Pavas Kant

Nathaniel White

Yuxuan Hao

Bonan Zheng

Yohan Jo

David A Jara

Xinyan Zhao

Nikolaos Malandrakis

Jiahui Li

Andrew Morris Werchniak

Aansh Shah

Nishtha Gupta

Raphael Petegrosso

Cody Jacques

Stefan M Bradstreet

A system performs federated learning and retraining of a machine learning model used for processing audio detected by a user device. The system uses both gradient data (which may correspond to false-rejects) and audio data (which may correspond to false-positives) received from devices. The system may also use a teacher model to produce labels for data in an automated fashion, thus allowing retraining to happen in an unsupervised manner.

Federated learning for audio processing

A device is configured to detect multiple different wakewords. A device may operate a joint encoder that operates on audio data to determine encoded audio data. The device may operate multiple different decoders which process the encoded audio data to determine if a wakeword is detected. Each decoder may correspond to a different wakeword. The decoders may use fewer computing resources than the joint encoder, allowing for the device to more easily perform multiple wakeword processing. Enabling/disabling wakeword(s) may involve the reconfiguring of a wakeword detector to add/remove data for respective decoder(s). Specific decoders may be activated/deactivated depending on device context, thereby efficiently managing device resources.

Audio detection

Techniques for determining when speech is directed at another individual of a dialog, and storing a representation of such user-directed speech for use as context when processing subsequently-received system-directed speech are described. A system receives audio data and/or video data and determines therefrom that speech in the audio data is user-directed. Based on this, the system determine whether the speech is able to be used to perform an action by the system. If the speech is able to be used to perform an action, the system stores a natural language representation of the speech. Thereafter, when the system receives system-directed speech, the system generates a rewrite of a natural language representation of the system-directed speech based on the previously-received user-directed speech. The system then determines output data responsive to the system-directed speech using the rewritten natural language representation.

Natural language generation

A system configured to perform content recognition using fingerprinting to recognize known media content. A device determines fingerprints based on decoded content data to be sent using a media interface component to an output component. Metadata related to the content/device/fingerprint may also be created. The fingerprints and metadata are sent by the device to a supporting system for orchestration and matching of the fingerprints to known media content.

Content recognition using fingerprinting

Systems and methods for preemptive wakeword detection are disclosed. For example, a first part of a wakeword is detected from audio data representing a user utterance. When this occurs, on-device speech processing is initiated prior to when the entire wakeword is detected. When the entire wakeword is detected, results from the on-device speech processing and/or the audio data is sent to a speech processing system to determine a responsive action to be performed by the device. When the entire wakeword is not detected, on-device processing is canceled and the device refrains from sending the audio data to the speech processing system.

Preemptive wakeword detection

A system that is capable of resolving anaphora using timing data received by a local device. A local device outputs audio representing a list of entries. The audio may represent synthesized speech of the list of entries. A user can interrupt the device to select an entry in the list, such as by saying 'that one.' The local device can determine an offset time representing the time between when audio playback began and when the user interrupted. The local device sends the offset time and audio data representing the utterance to a speech processing system which can then use the offset time and stored data to identify which entry on the list was most recently output by the local device when the user interrupted. The system can then resolve anaphora to match that entry and can perform additional processing based on the referred to item.

Dialog management for multiple users

Growing community of inventors

Cambridge, MA, United States of America

Aaron Challenner