Bose Corporation

Amazon Technologies, Inc.

Other

Wontak Kim

Laszlo Otto Drimusz

Mihir D Shetye

Michael J Daley

Anshuman Ganguly

Matthew S Walsh

Srivatsan Kandadai

Yuancheng Luo

Trausti Thor Kristjansson

Jahn Dmitri Eichfeld

Donna Marie Sullivan

Guy A Torio

George Nichols

George E P Chute

Joseph A Coffey, Jr

Zhen Sun

Austin Mackey

Mrudula V Athi

Mohamed Farouk Mansour

Michael Scott Dublin

Hal P Greenberger

Charles Oswald

Tobe Zetelmo Barksdale

Davis Pan

Michael W Stark

William M Rabinowitz

Joseph Jankovsky

Spencer Russell

Paraskevas Argyropoulos

Guangdong Pan

Lakshmi Venkatraman

Andrew Jackson Stockton X

Md Tamzeed Islam

Alexander M Epstein

Mahathir Monjur

Chaitanya Desai

Amod Kant Agrawal

Maurizio Bocca

James Michael Rush

Chad Jackman

Bulent Goksel

Davis Y Pan

George EP Chute

A device capable of using image data for purposes of determining a location of a user and audio beam selection to isolate audio in the direction of the user. The beamforming/beam-steering may occur after determining the user's location in order to conserve computing resources that would otherwise have been spent determining beams for non-desired directions. The beamformed audio may be used for speech processing, a communication session involving the device, or other purposes.

Beamforming using image data

Disclosed are various embodiments for localizing smart devices in a structure. An image of a portion of at least one room of a structure may be rendered on a display of the mobile device. A point in the image may be determined where a nonlocalized device is depicted, the nonlocalized device being positioned in the at least one room of the structure. The nonlocalized device is localized in the at least one room of the structure based on a position and orientation of the mobile device relative to the at least one room of the structure and based on a location of the point in the image. Alternatively, a location of a nonlocalized device in the structure may be determined by using the location of a mobile device as proxy.

Localization of devices in a structure

A wearable audio output device (e.g., headphones) having an open design that allows ambient noise to pass to a listener without physically isolating the listener from a surrounding environment. The device may include an open earcup design that may partially or completely surround the listener's ear, and in some examples a portion of the listener's head may be uncovered by the open earcup. To improve comfort, the device includes a floating audio component configured to generate output audio in a direction of the listener's ear without contacting the listener's ear. To reduce audio leakage into the environment, the floating audio component may include two audio transducers and perform beamforming to target the output audio toward the listener's ear and cancel the output audio in other directions. For example, the beamforming may cause constructive interference in one direction and destructive interference in all other directions.

Beamforming output audio using open headphones

A device capable of using data from multiple sensors to determine an estimated position/direction of a user with respect to the device. The device may use estimated position data, along with confidence data, that originated from a plurality of sensors to fuse the data to determine the user's estimated position and comprehensive confidence of the estimated position. The system may use the location information to perform beamforming/beam steering and/or other downstream operations using the comprehensive estimated position.

Beamforming using multiple sensor data

A system configured to perform user orientation estimation to determine a direction a user is facing using a deep neural network (DNN). As a directionality of human speech increases with frequency, the DNN may estimate the user orientation by comparing high-frequency components detected by each of the multiple devices. For example, a group of devices may individually generate feature data, which represents audio features and spatial information, and send the feature data to the other devices. Thus, each device in the group receives feature data generated by the other devices and processes this feature data using a DNN to determine an estimate of user orientation. In some examples, the DNN may also generate sound source localization (SSL) data and/or a confidence score associated with the user orientation estimate. A post-processing step may process the individual user orientation estimates generated by the individual devices and determine a final user orientation estimate.

User orientation estimation

A wearable audio output device (e.g., headphones) having an open design that allows ambient noise to pass to a listener without physically isolating the listener from a surrounding environment. The device may include an open earcup design that may partially or completely surround the listener's ear, and in some examples a portion of the listener's head may be uncovered by the open earcup. To improve comfort, the device includes a floating audio component configured to generate output audio in a direction of the listener's ear without contacting the listener's ear. To reduce an amount of ambient noise, the device may be configured to perform active noise cancellation (ANC) processing using feedforward and/or feedback microphones. The device may include an acoustic structure configured to direct the output audio in the direction of the listener's ear and/or position the feedback microphone(s) closer to the listener's ear.

Open headphones with active noise cancellation

Processes and devices for equalizing an audio system that is adapted to use a loudspeaker to transduce test audio signals into test sounds. The processes and devices can involve the use of infrared signals to convey information in one or both directions between the audio system and a portable computer device that captures test sounds, calculates audio parameters that can be used in the equalization process, and transmits these audio parameters back to the audio system for its use in equalizing audio signals that are played by the audio system.

Audio system equalizing

Disclosed are techniques for performing ray-based acoustic modeling that models scattering of acoustic waves by a surface of a device. The acoustic modeling uses two parameters, a room response representing acoustics and geometry of a room and a device response representing acoustics and geometry of the device. The room response is determined using ray-based acoustic modeling, such as ray tracing. The device response can be measured in an actual environment or simulated and represents an acoustic response of the device to individual acoustic plane waves. The device applies a superposition of the room response and the plane wave scattering from the device response to determine acoustic pressure values and generate microphone audio data. The device can estimate room impulse response (RIR) data using data from the microphones, and can use the RIR data to perform audio processing such as sound equalization, acoustic echo cancellation, audio beamforming, and/or the like.

Surface augmented ray-based acoustic modeling

Techniques for improving sound source localization (SSL) are provided. A method for probabilistic SSL using a deep neural network (DNN) may include receiving audio data including a representation of audio such as a wakeword from a microphone array. The audio data may be processed by a DNN to output a plurality of values where each value indicates a probability that the audio originated from a direction corresponding to that value. A sensor may provide computer vision or other data which may be used to inform the plurality of values based on detecting presence of a human or obstacle. A probability that the audio originated from one of the directions of the plurality of directions may be determined based at least in part on the DNN output and the computer vision or other data.

Sound source localization

A voice-controlled device includes a beamformer for determining audio data corresponding to one or more directions and a beam selector for selecting in which direction a source of target audio lies. The device determines magnitude spectrums for each beam and for each frequency bin in each beam for each frame of audio data. The device determines frame-by-frame changes in the magnitude and filters the changes to smooth them. The device selects the beam having the greatest smoothed change in magnitude as corresponding to speech.

Audio beam selection

A system configured to improve spatial coverage of output audio and a corresponding user experience by performing upmixing and loudspeaker beamforming to stereo input signals. The system can perform upmixing to the stereo (e.g., two channel) input signal to extract a center channel and generate three-channel audio data. The system may then perform loudspeaker beamforming to the three-channel audio data to enable two loudspeakers to generate output audio having three distinct beams. The user may interpret the three distinct beams as originating from three separate locations, resulting in the user perceiving a wide virtual sound stage despite the loudspeakers being spaced close together on the device.

Loudspeaker beamforming for improved spatial coverage

A beamformer system isolates a desired direction of an audio signal received from a first microphone array disposed on a first plane of the system and a second microphone array disposed on a second plane of the system. A spatial covariance matrix (SCM) defines the spatial covariance between pairs of microphones. A diagonal of the SCM is varied based on the placement of the microphones; values corresponding to one microphone array are increased, and values corresponding to the other microphone array are decreased.

Multi-plane microphone array

An omni-directional speaker system includes a deflector sub-assembly and a pair of acoustic sub-assemblies. The deflector sub-assembly includes a pair of diametrically opposed acoustic deflectors. Each of the acoustic sub-assemblies includes an acoustic driver for radiating acoustic energy toward an associated one of the acoustic deflectors. The acoustic sub-assemblies are coupled together via the deflector sub-assembly.

Omni-directional speaker system and related devices and methods

Growing community of inventors

Watertown, MA, United States of America

Wontak Kim