Amazon Technologies, Inc.

Berkant Tacer

Carlo Murgia

Alexander Kanaris

Gurhan Saplakoglu

Wai Chung Chu

Carlos Renato Nakagawa

Nikhil Shankar

Andreas Schwarz

Kanthasamy Chelliah

Shobha Devi Kuruba Buchannagari

Guillermo Daniel Garcia

Mazher Ali Khan

A system configured to perform source classification using a continuously active beamformer and a classifier to improve beamforming. A device processes audio data representing sounds from multiple sound sources to determine a source direction corresponding to each detected sound source. For each audio frame, the classifier receives a source direction for each unknown sound source and uses the source direction to determine first spectral characteristics that are unique to the individual sound source. By comparing the first spectral characteristics to spectral characteristics associated with labeled sound sources, the device identifies a match and associates the source directions with a corresponding labeled sound source. For each labeled sound source, the classifier determines attributes of a corresponding signal, and the device uses these attributes to select a single sound source. Using a desired look direction associated with the selected sound source, the beamformer generates audio data representing desired speech.

Sound source classification and beamforming

A system configured to perform noise dependent volume control in order to increase a volume level of the playback audio in a noisy environment. For example, a device may adaptively increase gain based on an amount of ambient noise present in the environment. The device may determine a gain value based on a noise reference value and an estimated noise floor. The device may determine the estimated noise floor based on continuous noises, ignoring transient sounds and user speech. In addition, the device may include volume control logic to ensure that the volume level only increases when necessary and/or to control a rate at which the volume level increases. As part of selecting the volume level with which to generate the playback audio, the device may use a lookup table to convert between a device volume and a quantized volume used to perform noise dependent volume control.

Noise dependent volume control

Maintaining time synchronization between clocks of different devices allows various operations, such as accurate timestamping input data, coordinating presentation of output, and so forth. A pairwise synchronization between a first and second device is provided by opportunistically including timing data within packets transmitted between the devices. A clock sync module in a second device uses a software-implemented phase-locked loop (PLL) with a software-implemented proportional-integral-derivative (PID) controller to synchronize a second clock in the second device to a first clock of the first device. The use of the clock sync module to process the timing data provides clock synchronization without introducing jitter or other undesirable effects at the second device. Using the clock sync module, the timing of the second clock of the second device quickly converges with the first clock. The close convergence facilitates time-sensitive operations, such as synchronizing presentation of content from many participating wireless devices.

System for time synchronization

A system configured to improve a voice quality during a communication session by performing bandwidth extension on a narrowband speech signal to generate a wideband speech signal with higher audio quality. For example, a system can extend a speech bandwidth from a narrowband signal having a first bandwidth (e.g., 4 kHz) to a wideband signal having a second bandwidth (e.g., 8 kHz or higher). To perform bandwidth extension, the system may include cascaded neural networks, such as two or more sub-pixel convolutional neural networks (CNNs) connected in series. In some examples, a first sub-pixel CNN may extend the speech bandwidth from 4 kHz to 6 kHz and a second sub-pixel CNN may extend the speech bandwidth from 6 kHz to 8 kHz. Alternatively, the system may use three or more cascaded neural networks and/or may extend the speech bandwidth above 8 kHz without departing from the disclosure.

Speech signal bandwidth extension using cascaded neural networks

A group of devices acquire audio input of a sound, such as speech, using respective microphones. For pairs of devices in the group, intensity of energy of audio input at each of the devices in the pair is used to determine first proximity data. Relative differences in time-of-arrival of the sound at the devices in the pair is used to determine second proximity data. The first and second proximity data are used to determine an estimated closest device of the pair with respect to the sound. Comparison of the first proximity data to the second proximity also allows a confidence value to be associated with the estimated closest device. The estimated closest device with the greatest confidence value may be selected for use to acquire audio input, present output, and so forth. Additional techniques such as beamforming techniques may be applied to the audio input from the selected device.

System to select audio from multiple connected devices

A system configured to perform distributed echo cancellation processing to attenuate feedback echo from occurring when two devices are acoustically coupled during a communication session. To reduce the feedback echo, one of the devices is configured as a hub device and receives microphone signals, synchronizes the microphone signals, and generates a mixed microphone signal. To enable distributed echo cancellation, the system includes bidirectional feedback link(s) between the hub device and each device synchronized with the hub device. For example, a first bidirectional feedback link sends a microphone signal from a second device to the hub device and sends the mixed microphone signal from the hub device to the second device, which the second device uses to perform echo cancellation. In addition, a second bidirectional feedback link sends a playback signal from the hub device to the second device and sends the output of echo cancellation back to the hub device.

Distributed feedback echo cancellation

A system configured to perform low input-output latency noise reduction in a frequency domain is provided. The real-time noise reduction algorithm performs frame by frame processing of a single-channel noisy acoustic signal to estimate a gain function. Accurate noise power estimates are achieved with the help of minimum statistics approach followed by a voice activity detector. The noise power and gain values are smoothed to remove any external artifacts and avoid background noise modulations. The gain values for individual frequency bands are weighted and smoothed to reduce distortion. To obtain distortionless output speech, the system performs curve fitting by separating the frequency bands into multiple groups and applying a Savitzky-Golay filter to each group. The final gain values generated by these filters are multiplied with the noisy speech signal to obtain a clean speech signal.

Spectral smoothing method for noise reduction

A system configured to improve audio processing by performing dereverberation and noise reduction during a communication session. The system may apply a two-channel dereverberation algorithm by calculating coherence-to-diffuse ratio (CDR) values and calculating dereverberation (DER) gain values based on the CDR values. While the DER gain values may be calculated at a first stage within the pipeline, the device may apply the DER gain values at a second stage within the pipeline. For example, the device may calculate the DER gain values prior to performing residual echo suppression (RES) processing but may apply the DER gain values after performing RES processing, in order to avoid excessive attenuation of the local speech. In addition to removing reverberation, the DER gain values also remove diffuse noise components, reducing an amount of noise reduction required. Thus, the device may soften noise reduction when the DER gain values are applied.

Dereverberation and noise reduction

A system configured to improve audio processing by performing dereverberation and noise reduction during a communication session. The system may apply a two-channel dereverberation algorithm by calculating coherence-to-diffuse ratio (CDR) values and calculating dereverberation (DER) gain values based on the CDR values. While the device calculates the DER gain values prior to performing acoustic echo cancellation (AEC) processing, the device applies the DER gain values after performing residual echo suppression (RES) processing in order to avoid excessive attenuation of the local speech. To improve output speech quality, the device does not apply the DER gain values for nonreverberant signals, when a signal-to-noise ratio (SNR) value is too low, and/or when far-end talk (e.g., remote speech) is present. Dereverberation processing is further improved by using frequency dependent parameters to calculate the DER gain values and by adjusting other gain values when the DER gain values are applied.

A multi-channel acoustic echo cancellation (AEC) system that includes a residual echo suppressor (RES) that dynamically controls an amount of attenuation to reduce distortion of local speech during double-talk conditions. The RES determines when double-talk conditions are present based on an echo return loss enhancement (ERLE) value. When the ERLE value is above a first threshold value but below a second threshold value, the RES reduces an amount of attenuation applied while generating an RES mask to pass local speech without distortion. When the ERLE value is below the first threshold value or above the second threshold value, the RES applies full attenuation while generating the RES mask in order to suppress a residual echo signal. To further improve RES processing, the RES may apply smoothing across time, smoothing across frequencies, or apply extra echo suppression processing to further attenuate the residual echo signal.

Growing community of inventors

Bellevue, WA, United States of America

Berkant Tacer