Amazon Technologies, Inc.

Ntt Docomo, Inc.

Rawles LLC

Other

Aod Technology Marketing, LLC

Wai Chung Chu

Carlo Murgia

Amit Singh Chhetri

Kavitha Velusamy

Ramya Gopalan

Edward Dietz Crump

Samuel Henry Chang

Hyeong Cheol Kim

Philip Ryan Hilmes

Robert Ayrapetian

Carlos Renato Nakagawa

Berkant Tacer

Khosrow Lashkari

Kanthasamy Chelliah

Andreas Schwarz

Wei Ey Li

Trausti Thor Kristjansson

Kuan-Chieh Yen

Qiang Liu

Ning Yao

Yuwen Su

Colter Earl Cederlof

Sowmya Gopalan

Manika Puri

Rahul Agrawal

Anshuman Ganguly

Borham Lee

Steve Anthony Quento

Steve Gil Gonzalez

John Walter McDonough, Jr

Robert Gregory Deacon

David J Pattridge

Le Quan Luong

A system that performs wall direction estimation to determine a position of an acoustically reflective surface relative to a device. For example, the device may detect a direct sound received from an active sound source and a first reflection reflected from a nearby wall. Based on a time delay, the device can estimate a direction and/or distance to the wall. For example, the device determines a time differential of arrival (TDOA) between the direct sound and the first reflection for each microphone in a microphone array. The device combines TDOA values of the entire microphone array into a distance vector, which the device can compare to reference distance data to determine the best wall direction estimate. For example, reference distance values are fixed based on a geometry of the microphone array and the device can identify an azimuth value that is most similar to the distance vector.

Method for wall direction estimation

A system configured to improve sound source localization (SSL) processing by including distance information. By estimating a 3D location vector corresponding to a direction and a distance associated with a sound source, the system may extend SSL processing from 2D to 3D and generate a location estimate for the sound source. In addition, the system groups the 3D location vectors into 3D location cells having a regular structure (e.g., predetermined uniformity and/or symmetry), which simplifies SSL processing and results in a substantial reduction in computational cost. For example, the system may associate each 3D location cell with (i) location vectors contained within the 3D location cell, (ii) weight values indicating a relative weight of each location vector for the 3D location cell, and (iii) neighboring 3D location cells. Using these associations, the system may identify peak power value(s) corresponding to sound source(s) and generate location estimates for each sound source.

Location estimation of sound sources

A system configured to perform sound source localization (SSL) using reflection detection is provided. A device processes audio data from multiple microphones to determine timing information corresponding to sound sources. For example, the device may determine cross-correlation data for each microphone pair, determine autocorrelation data for each microphone, and then use the autocorrelation data and the cross-correlation data to calculate quality factors. The device may determine the direction of potential sound source(s) by generating Steered Response Power (SRP) data using the cross-correlation data. To perform reflection detection to distinguish between direct sounds and acoustic reflections, the device may generate modified SRP data using the quality factors. For example, the device may process the SRP data to detect two potential sound sources and then process the modified SRP data to determine that a first potential sound source corresponds to direct sound and a second potential sound source corresponds to acoustic reflections.

Sound source localization with reflection detection

A system efficiently selects at least one device from multiple devices based on received audio signals. In some instances, the system receives audio signals from devices that each comprise at least one microphone. A respective audio signal of the audio signals includes a representation of a sound originating from a location. The system then determines a device to be used to respond to the sound. In some instances, the system analyzes times in which the received audio signals that represent the sound are generated and/or volumes of the sound as represented by the received audio signals. The system can then select the device based on the analysis.

Analyzing audio signals for device selection

A system configured to improve sound source localization (SSL) processing by reducing a number of direction vectors and grouping the direction vectors into direction cells is provided. The system performs clustering to generate a smaller set of direction vectors included in a delay-direction codebook, reducing a size of the codebook to the number of unique delay vectors. In addition, the system groups the direction vectors into direction cells having a regular structure (e.g., predetermined uniformity and/or symmetry), which simplifies SSL processing and results in a substantial reduction in computational cost. The system may also select between multiple codebooks and/or dynamically adjust the codebook to compensate for changes to the microphone array. For example, a device with a microphone array fixed to a display that can tilt may adjust the codebook based on a tilt angle of the display to improve accuracy.

Direction finding of sound sources

A system configured to improve track selection while performing audio type detection using sound source localization (SSL) data is provided. A device processes audio data representing sounds from multiple sound sources to determine SSL data that distinguishes between each of the sound sources. The system detects an acoustic event and performs SSL track selection to select the sound source that corresponds to the acoustic event based on input features. To improve SSL track selection, the system detects current conditions of the environment and determines adaptive weight values that vary based on the current conditions, such as a noise level of the environment, whether playback is detected, whether the device is located near one or more walls, etc. By adjusting the adaptive weight values, the system improves an accuracy of the SSL track selection by prioritizing the input features that are most predictive during the current conditions.

Sound source localization

A system configured to perform directional speech separation. The system may dynamically associate direction-of-arrivals with one or more audio sources in order to generate output audio data that separates each of the audio sources. The system identifies a target direction for each audio source, dynamically determines directions that are correlated with the target direction, and generates output signals for each audio source. The system may associate individual frequency bands with specific directions based on a time delay detected by two or more microphones. The system may determine a cross-correlation between each direction and the target direction and select directions with strong correlation. The system may generate time-frequency mask data indicating frequency bands corresponding to the directions associated with a particular audio source. Using the mask data, the system generates output audio data specific to the audio source, resulting in directional speech separation between different audio sources.

Directional speech separation

A system configured to perform sound source localization (SSL) using reflection classification is provided. A device processes audio data representing sounds from multiple sound sources to generate sound track data that includes an individual sound track for each of the sound sources. To detect reflections, the device determines whether a pair of sound tracks are strongly correlated. For example, the device may calculate a correlation value for each pairwise combination of the sound tracks and determine whether the correlation value exceeds a threshold value. When the correlation value exceeds the threshold, the device invokes a reflection classifier trained to distinguish between direct sound sources and reflected sound sources. For example, the device extracts feature data from the pair of sound tracks and processes the feature data using a trained model to determine which of the sound tracks corresponds to the direct sound source.

Sound source localization using reflection classification

A system configured to improve audio processing by performing dereverberation and noise reduction during a communication session. The system may apply a two-channel dereverberation algorithm by calculating coherence-to-diffuse ratio (CDR) values and calculating dereverberation (DER) gain values based on the CDR values. While the DER gain values may be calculated at a first stage within the pipeline, the device may apply the DER gain values at a second stage within the pipeline. For example, the device may calculate the DER gain values prior to performing residual echo suppression (RES) processing but may apply the DER gain values after performing RES processing, in order to avoid excessive attenuation of the local speech. In addition to removing reverberation, the DER gain values also remove diffuse noise components, reducing an amount of noise reduction required. Thus, the device may soften noise reduction when the DER gain values are applied.

Dereverberation and noise reduction

A system configured to improve audio processing by performing dereverberation and noise reduction during a communication session. The system may apply a two-channel dereverberation algorithm by calculating coherence-to-diffuse ratio (CDR) values and calculating dereverberation (DER) gain values based on the CDR values. While the device calculates the DER gain values prior to performing acoustic echo cancellation (AEC) processing, the device applies the DER gain values after performing residual echo suppression (RES) processing in order to avoid excessive attenuation of the local speech. To improve output speech quality, the device does not apply the DER gain values for nonreverberant signals, when a signal-to-noise ratio (SNR) value is too low, and/or when far-end talk (e.g., remote speech) is present. Dereverberation processing is further improved by using frequency dependent parameters to calculate the DER gain values and by adjusting other gain values when the DER gain values are applied.

A device capable of autonomous motion may move in response to a user speaking an utterance, such as a command. Before moving, the device processes audio data received from a microphone array to identify different audio signals arriving at the device from different directions. Based on properties of the audio signals, the device determines which of the audio signals are merely reflections of other audio.

Autonomously motile device with audio reflection detection

A system configured to perform directional speech separation using three or more microphones. The system may dynamically associate direction-of-arrivals with one or more audio sources in order to generate output audio data that separates each of the audio sources. Using three or more microphones, the system may separate audio sources covering 360 degrees surrounding the microphone array, whereas a two-microphone implementation is limited to 180 degrees. The system identifies a target direction for each audio source, dynamically determines directions that are correlated with the target direction, and generates output signals for each audio source. The system may associate individual frequency bands with specific directions based on a phase difference detected by two or more microphones.

Omni-directional speech separation

An augmented reality environment allows interaction between virtual and real objects. Multiple microphone arrays of different physical sizes are used to acquire signals for spatial tracking of one or more sound sources within the environment. A first array with a larger size may be used to track an object beyond a threshold distance, while a second array having a size smaller than the first may be used to track the object up to the threshold distance. By selecting different sized arrays, accuracy of the spatial location is improved.

Sound source localization using multiple microphone arrays

This disclosure describes, in part, techniques for processing audio data. For instance, an electronic device may include an automatic gain controller (AGC) that determines AGC gains for amplifying or attenuating an audio data. To determine the AGC gains, the AGC uses information from a residual echo suppressor (RES) and/or a noise reductor (NR). The information may indicate RES gains applied to the audio data by the RES and/or NR gains applied to the audio data by the NR. In some instances, to determine the AGC gain, the AGC determines time-constant parameter(s) using the information. The AGC then uses the time-constant parameter(s) to determine an input signal level for the audio data and/or the AGC gain. In some instances, to determine the AGC gain, the AGC operates in an attack mode or a release mode based on the information.

Growing community of inventors

San Jose, CA, United States of America

Wai Chung Chu