Amazon Technologies, Inc.

Audience, Inc.

Knowles Electronics, Inc.

Mindspeed Technologies, Inc.

Invensense, Inc.

Sensor Platforms, Inc.

Carlo Murgia

Carlos M Avendano

Ludger Solbach

Ye Jiang

Wai Chung Chu

Mark Every

Kuan-Chieh Yen

Berkant Tacer

Eyal Shlomot

Michael M Goodwin

Adil Benyassine

Philip Ryan Hilmes

Robert Ayrapetian

David Klein

Peter Santos

Ajay Kumar Dhanapalan

Jeffrey D Klein

Taro Kimura

Yang Gao

Dana C Massie

Carlos Renato Nakagawa

Alex Afshar

Huan-Yu Su

Mohamed Farouk Mansour

William Kerry Keal

Jean Laroche

Trausti Thor Kristjansson

Ramya Gopalan

Mark Robert Every

Eitan Medina

Aditya Sharadchandra Joshi

Sangnam Choi

Deborah Kathleen Vitus

James V Steele

Michael Thomas Peterson

Zhouhui Miao

Hyeong Cheol Kim

Kanthasamy Chelliah

Andreas Schwarz

Daniel Wayne Harris

Karim Younes

Scott Isabelle

Anthony D Minervini

Shobha Devi Kuruba Buchannagari

Milos Jorgovanovic

James Robert Lim

Ke Li

Fariborz Assaderaghi

Tarun Pruthi

Rong Hu

Brian D Clark

Lloyd Watts

Alexander Kanaris

Sridhar Pilli

Paraskevas Argyropoulos

Dibyendu Nandy

Benjamin E Joseph

Kevin A Shaw

Ian Ernan Liu

Sridhar Krishna Nemala

Anshuman Ganguly

Borham Lee

Frederic Caldwell

Madhuri Saraf

Guillermo Daniel Garcia

Alex Kanaris

Matthew Cowan

Aruna Vittal

Huiqun Han

Kunal Atitkar

Mazher Ali Khan

Biqing Wu

Phil Hetherington

Eric Skup

Craig S Mellon

A system that performs wall direction estimation to determine a position of an acoustically reflective surface relative to a device. For example, the device may detect a direct sound received from an active sound source and a first reflection reflected from a nearby wall. Based on a time delay, the device can estimate a direction and/or distance to the wall. For example, the device determines a time differential of arrival (TDOA) between the direct sound and the first reflection for each microphone in a microphone array. The device combines TDOA values of the entire microphone array into a distance vector, which the device can compare to reference distance data to determine the best wall direction estimate. For example, reference distance values are fixed based on a geometry of the microphone array and the device can identify an azimuth value that is most similar to the distance vector.

Method for wall direction estimation

A system configured to perform noise dependent volume control in order to increase a volume level of the playback audio in a noisy environment. For example, a device may adaptively increase gain based on an amount of ambient noise present in the environment. The device may determine a gain value based on a noise reference value and an estimated noise floor. The device may determine the estimated noise floor based on continuous noises, ignoring transient sounds and user speech. In addition, the device may include volume control logic to ensure that the volume level only increases when necessary and/or to control a rate at which the volume level increases. As part of selecting the volume level with which to generate the playback audio, the device may use a lookup table to convert between a device volume and a quantized volume used to perform noise dependent volume control.

Noise dependent volume control

This disclosure describes, in part, techniques to process audio signals to lessen the impact that wind and/or other environmental noise has upon the resulting quality of audio signals output on speaker(s) of wireless earbuds or other headphones. For example, the techniques may determine a level of wind and/or other noise in an environment and may determine how best to process the signals to lessen the impact of the noise, such that one or more users that hear audio based on output of the signals hear higher-quality audio. In some instances, the techniques are able to detect levels of wind in a relatively low-power manner and as part of regular operation of the wireless earbuds or other headphones.

Mitigating effects of wind in audio data

A system configured to generate dynamic visual feedback by synchronizing smart lights with a rhythm and energy levels of music being output. A device may translate audio samples to luma (brightness), chroma (color), and location data using a low-latency algorithm. For example, the device may divide the full band audio data into separate frequency bands based on the number of output devices and control a brightness for an output device by tracking energy within a corresponding frequency band. The color and location may be controlled based on a number of beats per minute, direction, and/or the like. In addition to controlling external lighting via a smart home hub, the device can control LED light(s) on the device via an LED controller. To further reduce latency, the device can directly communicate between an audio processor, the smart home hub, and/or the LED controller using low level hardware abstraction layers (HALs).

Accessory control using smart speakers

A system configured to perform audio type detection for sound source localization (SSL) data is provided. A device processes audio data representing sounds from multiple sound sources to determine SSL data that distinguishes between each of the sound sources. To identify an audio type associated with a sound source and/or track individual sound sources over time, the device can determine a correlation between the SSL data and an audio event. Examples of audio events include a wakeword component detecting a wakeword or an acoustic event detector detecting a particular acoustic event. The device may determine a correlation between wakeword data indicating that a wakeword is represented in the audio data and SSL data for each individual sound source. The device may then identify a sound source that is most strongly correlated to the wakeword data and associate the wakeword and a corresponding voice command with the sound source.

Sound source localization audio type detection

A system configured to use keywords to augment audio playback with visual effects and/or other effects to provide an immersive audio experience. For example, a device can detect a keyword and control a color and intensity of external lights to provide visual feedback. In addition to visual effects, the device can trigger additional effects using smart plugs or other smart devices. In a listening enhancement mode in which the device outputs audio content, the device performs keyword detection by monitoring playback audio data for preconfigured keywords. In a storytelling mode in which a user reads a book out loud, the device may perform keyword detection by monitoring microphone audio data for the preconfigured keywords. Controlling the visual effects in response to keyword detection is enabled by a new pipeline that sends information back from a wakeword engine to an audio processor.

Accessory control using keywords

Maintaining time synchronization between clocks of different devices allows various operations, such as accurate timestamping input data, coordinating presentation of output, and so forth. A pairwise synchronization between a first and second device is provided by opportunistically including timing data within packets transmitted between the devices. A clock sync module in a second device uses a software-implemented phase-locked loop (PLL) with a software-implemented proportional-integral-derivative (PID) controller to synchronize a second clock in the second device to a first clock of the first device. The use of the clock sync module to process the timing data provides clock synchronization without introducing jitter or other undesirable effects at the second device. Using the clock sync module, the timing of the second clock of the second device quickly converges with the first clock. The close convergence facilitates time-sensitive operations, such as synchronizing presentation of content from many participating wireless devices.

System for time synchronization

A system configured to improve sound source localization (SSL) processing by reducing a number of direction vectors and grouping the direction vectors into direction cells is provided. The system performs clustering to generate a smaller set of direction vectors included in a delay-direction codebook, reducing a size of the codebook to the number of unique delay vectors. In addition, the system groups the direction vectors into direction cells having a regular structure (e.g., predetermined uniformity and/or symmetry), which simplifies SSL processing and results in a substantial reduction in computational cost. The system may also select between multiple codebooks and/or dynamically adjust the codebook to compensate for changes to the microphone array. For example, a device with a microphone array fixed to a display that can tilt may adjust the codebook based on a tilt angle of the display to improve accuracy.

Direction finding of sound sources

Techniques for encoding audio data with metadata are described. In an example, a device receives audio data corresponding to audio detected by a microphone and receives metadata associated with the audio. The device generates encoded data based at least in part on encoding the audio data with the metadata. The encoding involves replacing a portion of the audio data with the metadata, such that the encoded data includes the metadata and a remaining portion of the audio data. The device sends the encoded data to an audio processing application.

Encoding audio metadata in an audio frame

Techniques for improving adaptive interference cancellation (AIC) using cascaded AIC algorithms are described. To improve an accuracy of detecting speech, a device may perform a first stage of AIC to generate isolated audio data and may generate speech mask data indicating time windows when speech is detected in the isolated audio data. Based on the speech mask data, the device may perform second AIC to generate output audio data, with adaptation of the adaptive filter enabled when the speech is not detected and disabled when the speech is detected. Thus, the first AIC improves the accuracy with which the device detects that speech is present and the second AIC reduces distortion in the output audio data by not updating filter coefficient values when the speech is present. The first AIC may use playback audio data, microphone audio data or beamformed audio data as reference signals.

Cascaded adaptive interference cancellation algorithms

This disclosure describes techniques for detecting voice commands from a user of an ear-based device. The ear-based device may include an in-ear facing microphone to capture sound emitted in an ear of the user, and an exterior facing microphone to capture sound emitted in an exterior environment of the user. The in-ear microphone may generate an inner audio signal representing the sound emitted in the ear, and the exterior microphone may generate an outer audio signal representing sound from the exterior environment. The ear-based device may compute a ratio of a power of the inner audio signal to the outer audio signal and may compare this ratio to a threshold. If the ratio is larger than the threshold, the ear-based device may detect the voice of the user. Further, the ear-based device may set a value of the threshold based on a level of acoustic seal of the ear-based device.

Voice detection using ear-based devices

A system configured to perform microphone occlusion event detection. When a device detects a microphone occlusion event, the device will modify audio processing performed prior to speech processing, such as by disabling spatial processing and only processing audio data from a single microphone. The device detects the microphone occlusion event by determining inter-level difference (ILD) values between two microphone signals and using the ILD values as input features to a classifier. For example, when a far-end reference signal is inactive, the classifier may process a first ILD value within a high frequency band. However, when the far-end reference signal is active, the classifier may process the first ILD value and a second ILD value within a low frequency band.

Microphone occlusion detection

A system configured to improve audio processing by performing dereverberation and noise reduction during a communication session. The system may apply a two-channel dereverberation algorithm by calculating coherence-to-diffuse ratio (CDR) values and calculating dereverberation (DER) gain values based on the CDR values. While the DER gain values may be calculated at a first stage within the pipeline, the device may apply the DER gain values at a second stage within the pipeline. For example, the device may calculate the DER gain values prior to performing residual echo suppression (RES) processing but may apply the DER gain values after performing RES processing, in order to avoid excessive attenuation of the local speech. In addition to removing reverberation, the DER gain values also remove diffuse noise components, reducing an amount of noise reduction required. Thus, the device may soften noise reduction when the DER gain values are applied.

Dereverberation and noise reduction

Techniques for detecting a voice command from a user of a hearable device. The hearable device may include an in-ear facing microphone to capture sound emitted from an ear of the user, and an exterior facing microphone to capture sound emitted from an exterior environment of the user. The in-ear microphone may generate an in-ear audio signal representing the sound emitted from the ear, and the exterior microphone may generate an exterior audio signal representing sound from the exterior environment. The hearable device may include components to determine correlations or similarities between the in-ear audio signal and exterior audio signal, which indicate that the audio signals represent sound emitted from the user. Further, the components may perform voice activity detection to determine that the sound emitted from the user is a voice command, and proceed to perform further voice-processing techniques.

Voice detection using hearable devices

Growing community of inventors

Sunnyvale, CA, United States of America

Carlo Murgia