Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Rawles LLC

Other

Amit Singh Chhetri

Ramya Gopalan

Kavitha Velusamy

Philip Ryan Hilmes

Wai Chung Chu

William Folwell Barton

Navin Chatlani

Trausti Thor Kristjansson

Srivatsan Kandadai

Kenneth Edgar Hild, Ii

Jeffrey C O'Neill

John Carlton Platt

Leo Benedict Baldwin

Omar Sze Leung

Noam Sorek

Arungunram Chandrasekaran Surendran

Shiva Kumar Sundaram

Jack W Stokes

Serkan Hatipoglu

David William Devries

Michael Devyver

Stewart Robin Shearer

Wei Ey Li

Mohamed Farouk Mansour

Dinei A Florencio

Rohit Prasad

Edward Dietz Crump

Michael Alan Pogue

Yue Liu

Robert Ayrapetian

Shiv Naga Prasad Vitaladevuni

Jack Wilson Stokes, Iii

Ludger Solbach

Rong Hu

Rui Wang

Xianxian Zhang

Charles Guest Tritschler

Hongyang Deng

Pradeep Kumar Govindaraju

Xiaoxue Li

Mrudula V Athi

John Walter McDonough, Jr

Md Tamzeed Islam

Ananth Raghavendra

Laurent Andre Ovaert

Praveen Jayakumar

Phillip Ryan Hilmes

Jack W Iii Stokes

Techniques for presence detection based on multi modal sensors are described. In an example, a computer system determines first data indicating a first prediction of presence detection within a space. The first data is generated based on first sensor data. The computer system determines second data indicating a second prediction of presence detection within the space. The second data is generated based on second sensor data. The second sensor data is of a different type than the first sensor data. The computer system generates third data indicating a third prediction of presence detection within the space based on the first data, the second data, and a fusion model. The fusion model is configured to generate the third data by at least using associations between presence detections, latent variables, and observed variables. The observed variables correspond to the first data and the second data. The latent variables include sensor-triggering events.

Presence detection based on multiple sensors

A system configured to improve audio processing by performing dereverberation and noise reduction during a communication session. In some examples, the system may include a deep neural network (DNN) configured to perform speech enhancement, which is located after an Acoustic Echo Cancellation (AEC) component. For example, the DNN may process isolated audio data output by the AEC component to jointly mitigate additive noise and reverberation. In other examples, the system may include a DNN configured to perform acoustic interference cancellation, which may jointly mitigate additive noise, reverberation, and residual echo, removing the need to perform residual echo suppression processing. The DNN is configured to process complex-valued spectrograms corresponding to the isolated audio data and/or estimated echo data generated by the AEC component.

Dereverberation and noise reduction

A device capable of autonomous motion may move in an environment and may receive audio data from a microphone. A model may be trained to process the audio data to suppress noise from the audio data. The model may include an encoder that includes one or more convolutional layers, one or more recurrent layers, and a decoder that includes one or more convolutional layers.

Autonomously motile device with noise suppression

A device capable of autonomous motion may move in an environment and may receive audio data from a microphone. A model may be trained to process the audio data to determine mask data, which may be used to mask noise in the audio data. Training data for the model may be normalized before training, and different loss functions may be used for different types of training data.

The systems, devices, and processes described herein may identify a beam of a voice-controlled device that is directed toward a reflective surface, such as a wall. The beams may be created by a beamformer. An acoustic echo canceller (AEC) may create filter coefficients for a reference sound. The filter coefficients may be analyzed to identify beams that include multiple peaks. The multiple peaks may indicate presence of one or more reflective surfaces. Using the amplitude and the time delay between the peaks, the device may determine that it is close to a reflective surface in a direction of the beam.

Beam rejection in multi-beam microphone systems

A computing device may receive audio data from a microphone representing audio in an environment of the device, which may correspond to an utterance and noise. A model may be trained to process the audio data to cancel noise from the audio data. The model may include an encoder that includes one or more dense layers, one or more recurrent layers, and a decoder that includes one or more dense layers.

Systems and methods for noise cancellation

A device capable of motion includes a beamformer for determining audio data corresponding to one or more directions. The beamformer includes a target beamformer that boosts audio from a target direction and a null beamformer that suppresses audio from that direction. When the device outputs sound while moving, the target and null beamformers capture and compensate for Doppler effects in output audio that reflects from nearby surfaces back to the device.

Autonomously motile device with adaptive beamforming

A device capable of autonomous motion includes a residual echo suppressor for suppressing echoes caused by an output reference signal. When the device outputs audio while moving with a velocity, it may receive echoes that are Doppler-shifted due to the motion. The residual echo suppressor generates estimated residual error data based on phase-shifted reference data to account for and suppress the Doppler-shifted echoes.

Autonomously motile device with residual echo suppression

A mobile device capable of capturing voice commands includes a beamformer for determining audio data corresponding to one or more directions and a beam selector for selecting in which direction a source of target audio lies. The device determines, based on data from one or more sensors, an angle through which the device has rotated. Based on the angle and one or more rotation-compensation functions, the device interpolates audio data corresponding to the one or more directions to compensate for the rotation such that the direction corresponding to the source of target audio remains selected.

Beamformer rotation

A device capable of motion includes an acoustic echo canceller for cancelling a reference signal from received audio data. The device updates an adaptive filter as the device moves to reflect the changing audio channel between a loudspeaker and a microphone of the device. A step size for changing coefficients of the filter is determined based on its velocity. A number of iterations for updating the filter using a frame of audio data is also determined based on the velocity.

Autonomously motile device with acoustic echo cancellation

A device capable of moving a component of the device motion is capable of determining audio data corresponding to a direction relative to the device ('beamforming'). When the component moves, the device determines a new position of the component and selects one of a set of beamforming filter coefficients corresponding to the position. Using the filter coefficients, the device determines the directional audio data.

Autonomously motile device with beamforming

A wearable computer is configured to use beamforming techniques to isolate a user's speech from extraneous audio signals occurring within a physical environment. A microphone array of the wearable computer may generate audio signal data from an utterance from a user's mouth. A motion sensor(s) of the wearable computer may generate motion data from movement of the wearable computer. This motion data may be used to determine a direction vector pointing from the wearable computer to the user's mouth, and a beampattern may be defined that has a beampattern direction in substantial alignment with the determined direction vector to focus the microphone array on the user's mouth for speech isolation.

Beamforming for a wearable computer

A decision engine executing on an electronic device may determine, using sensor data captured by multiple sensors of the device, whether a user is present in an environment that includes the device. If the user is determined to be present in the environment, the device may transition from a first state to a second state. The first state may be a first power state of the device in which the device is powered off or an idle or dormant state in which the device is powered on but a display of the device is powered off. Correspondingly, the second state may be a second power state of the device in which the device and the display are powered on and content is being rendered on the display. If the decision engine cannot make a determination based on the sensor data, a context engine may adjudicate the user presence determination.

Determining a device state based on user presence detection

Growing community of inventors

Santa Clara, CA, United States of America

Amit Singh Chhetri