Amazon Technologies, Inc.

Apple Inc.

Starkey Laboratories, Inc.

Tarun Pruthi

Trausti Thor Kristjansson

Krishna Kamath Koteshwara

Ali Abdollahzadeh Milani

Yuzhou Liu

Spencer Russell

Harsha Inna Kedage Rao

Savaskan Bulek

Joshua D Atkins

Ke Li

Vasu Iyengar

William S Woods

Rui Wang

Anran Wang

Aram M Lindahl

Carlo Murgia

Sean Anthony Ramprashad

Esge Brandt Andersen

Yacine Azmi

Sorin V Dusan

Ludger Solbach

Kuan-Chieh Yen

Srivatsan Kandadai

Ashrith Deshpande

Jakub Mazur

Minje Kim

Lalin S Theverapperuma

Wan-Chieh Pai

Jinjun Xiao

Maziyar Khorasani

Zhen Sun

Narimene Lezzoum

Alex Kanaris

Xavier Chabot

Lars Rasmus Elofsson Bernstedt

A system configured to perform neural sidelobe cancelling to achieve enhanced target speech separation. A device improves target speech separation by combining a fixed directional beamformer with a deep neural network (DNN) configured to (i) selectively enhance speech in one target direction, (ii) cancel speech from non-target directions, and (iii) cancel non-speech interference from all directions. By combining the directional output from the beamformer with raw microphone signals as input signals to the DNN, the DNN is given a directional cue with which to select the target speech. The DNN generates mask data representing the target speech using a combination of temporal convolutional blocks and causal self-attention blocks. For example, the DNN achieves low latency by processing a large number of short frames, while the causal self-attention blocks smooth the output and reinforce local consistency of speech.

Neural sidelobe canceller for target speech separation

A system that performs wall detection and localization to determine a position of a device relative to acoustically reflective surfaces. The device emits an audible sound including a frequency modulated signal and captures reflections of the audible sound. The frequency modulated signal enables the device to determine an amplitude of the reflections at different time-of-arrivals, which corresponds to a direction of the reflection. The device then performs beamforming to generate a 2D intensity map that represents an intensity of the reflections at each spatial location around the device. The device detects wall(s) in proximity to the device by identifying peak intensities represented in the 2D intensity map. In some examples, instead of performing beamforming, the device can perform directional wall detection by physically rotating the device and emitting the audible sound in multiple directions. The device may perform ultrasonic wall detection using ultrasonic sound frequencies.

Method for wall detection and localization

Techniques for presence-detection devices to vary presence-detection sensitivity to detect different types of movements by objects, such as major movements and minor movements, using ultrasonic signals. The devices detect movement of a person in an environment by emitting ultrasonic signals into the environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person relative to the presence-detection devices. The presence-detection devices may control the presence-detection sensitivity in order to detect major movements, such as a user walking in a room, as well as objects minor movements, such as a user typing at a computer. By adjusting the presence-detection sensitivity, the devices are able to improve the overall accuracy of detecting the presence of users in an environment.

Controlling sensitivity of presence detection using ultrasonic signals

Techniques for monitoring devices to use ultrasonic signals to detect and track the locations of moving objects in an environment. To determine distance information, the monitoring devices emit a frequency-modulated continuous wave (FMCW) signal at an ultrasound frequency range. Reflections of the FMCW ultrasonic signal are used to generate time-of-arrival (TOA) profiles that indicate distances between the monitoring device and objects in the environment. The reflections can be processed to suppress undesirable interferences, such as reflections off non-mobile objects in the environment (e.g., walls, furniture, etc.), vibrations off the floorings or the ceilings, etc. After processing the reflections, a heatmap can be used to plot the intensity of the reflections for the different TOAs of the reflections, and depict the movement of the user over time. Finally, a Kalman filter is used to smooth the peaks in the intensity values on the plot, and determine the trajectory of the human.

Monitoring non-stationary object distance using ultrasonic signals

A system configured to detect a tap event on a surface of a device only using microphone audio data. For example, instead of using a physical sensor to detect the tap event, the device may detect a tap event in proximity to a microphone based on a power level difference between two or more microphones. When a power ratio exceeds a threshold, the device may detect a tap event and perform an action. For example, the device may output an alarm and use a detected tap event as an input to delay or end the alarm. In some examples, the device may detect a tap event using a plurality of microphones. Additionally or alternatively, the device may distinguish between multiple tap events based on a location of the tap event, enabling the device to perform two separate actions depending on the location.

Inter-channel level difference based acoustic tap detection

A system configured to perform deep adaptive acoustic echo cancellation (AEC) to improve audio processing. Due to mechanical noise and continuous echo path changes caused by movement of a device, echo signals are nonlinear and time-varying and not fully canceled by linear AEC processing alone. To improve echo cancellation, deep adaptive AEC processing integrates a deep neural network (DNN) and linear adaptive filtering to perform echo and/or noise removal. The DNN is configured to generate a nonlinear reference signal and step-size data, which the linear adaptive filtering uses to generate output audio data representing local speech. The DNN may generate the nonlinear reference signal by generating mask data that is applied to a microphone signal, such that the reference signal corresponds to a portion of the microphone signal that does not include near-end speech.

Deep adaptive acoustic echo cancellation

A computing device may receive audio data from a microphone representing audio in an environment of the device, which may correspond to an utterance and noise. A model may be trained to process the audio data to reduce noise from the audio data. The model may include an encoder that includes one or more dense layers, one or more recurrent layers, and a decoder that includes one or more dense layers.

Systems and methods for noise reduction

Techniques for presence-detection devices to emission levels of ultrasonic signals that are used to detect movement in an environment. The presence-detection devices may detect movement of a person by emitting the ultrasonic signals into an environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person relative to the presence-detection devices. However, presence-detection devices that continuously emit ultrasonic signals may experience reduced battery life, increased likelihood of overheating, etc. To reduce these negative effects, the presence-detection devices may reduce the emission levels of ultrasonic signals. For instance, once motion is detected, the presence-detection devices may, for a period of time, stop emitting ultrasonic signals or reduce the power level at which the ultrasonic signals are emitted. Accordingly, the presence-detection devices can reduce emission levels of ultrasonic signals while still detecting motion.

Controlling emission of ultrasonic signals for presence detection

A personal listening system and a method of using the personal listening system to enhance speech intelligibility of audio playback, are described. The method includes determining a speech intelligibility metric, such as a speech reception threshold, of a user. Based on the speech intelligibility metric, a tuning parameter is applied to an audio input signal. The speech reception threshold is compared to an environmental signal-to-noise ratio to determine whether enhancement of the audio input signal is warranted. Application of the tuning parameter to the audio input signal generates an audio output signal having reduced noise, making playback of the audio output signal more intelligible to the user. Other aspects are also described and claimed.

System and method of enhancing intelligibility of audio playback

Techniques for presence-detection devices to detect movement of a person in an environment by emitting ultrasonic signals using a loudspeaker that is concurrently outputting audible sound. To detect movement by the person, the devices characterize the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. However, when a loudspeaker plays audible sound while emitting the ultrasonic signal, audio signals generated by microphones of the devices include distortions caused by the loudspeaker. These distortions can be interpreted by the presence-detection devices as indicating movement of a person when there is no movement, or as indicating lack of movement when a user is moving. The techniques include processing audio signals to remove distortions to more accurately identify changes in the frequency of the reflections of the ultrasonic signals caused by the movement of the person.

Presence detection using ultrasonic signals with concurrent audio playback

Techniques for calibrating presence-detection devices to account for various factors that can affect the presence-detection devices' ability to detect movement. Presence-detection devices may detect movement of a person in an environment by emitting ultrasonic signals into the environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. However, factors such as environmental acoustic conditions, noise sources, etc., may affect the ability of the presence-detection devices to detect movement. To calibrate for these factors, the presence-detection devices may use a loudspeaker to emit an ultrasonic sweep signal that spans different frequencies in an ultrasonic frequency range. The presence-detection devices may generate audio data using a microphone that represents the ultrasonic sweep signal, and analyze that audio data to determine an optimal frequency range and/or transmission power for subsequent ultrasonic signal transmissions.

Device calibration for presence detection using ultrasonic signals

This disclosure describes presence-detection devices that detect movement of a person by emitting ultrasonic signals into an environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. The techniques include downsampling the audio signals from the carrier frequency range down to a frequency range with a center frequency around 0 Hz. A filter is applied to attenuate signals around 0 Hz and below (or above), such as the emitted signals. In addition to removing the emitted signals, the negative side (or positive side) of the audio signals are removed, but the Doppler shift is still represented in the remaining portion of the audio signals. By removing a portion of the audio signals, the amount of processing required to detect the Doppler shift in the reflections of the ultrasonic signals is reduced.

Processing audio signals for presence detection

This disclosure describes, in part, techniques to process audio signals to lessen the impact that wind and/or other environmental noise has upon the resulting quality of these audio signals. For example, the techniques may determine a level of wind and/or other noise in an environment and may determine how best to process the signals to lessen the impact of the noise, such that one or more users that hear audio based on output of the signals hear higher-quality audio.

Processing audio to account for environmental noise

An audio system has a housing in which are integrated a number of microphones. A programmed processor accesses the microphone signals and produces a number of acoustic pick up beams based groups of microphones, an estimation of voice activity and an estimation of noise characteristics on each beam. Two or more beams including a voice beam that is used to pick up a desired voice and a noise beam that is used to provide information to estimate ambient noise are adaptively selected from among the plurality of beams, based on thresholds for voice separation and thresholds for noise-matching. Other embodiments are also described and claimed.

Growing community of inventors

Fremont, CA, United States of America

Tarun Pruthi