Amazon Technologies, Inc.

Spencer Russell

Tarun Pruthi

Krishna Kamath Koteshwara

Carlos Renato Nakagawa

Trausti Thor Kristjansson

Anran Wang

Mohamed Farouk Mansour

Shobha Devi Kuruba Buchannagari

Wontak Kim

Ali Abdollahzadeh Milani

Wan-Chieh Pai

Anshuman Ganguly

Yuzhou Liu

Zhen Sun

Mrudula V Athi

Fnu Anish Kumar

Alexander M Epstein

A system configured to create a flexible home theater group using a variety of different devices. To enable synchronized audio output, the system performs device localization to generate map data representing locations of the devices. The system determines distance values between devices using timing information generated during calibration even when the devices themselves may not have synchronized clocks. During calibration, each device will generate a calibration tone in a particular order, enabling listening devices to detect the calibration tone and determine a relative direction of the output device. The listening devices also generate timing information indicating when each calibration tone was detected which can be used to determine a propagation delay that corresponds to a distance between the output device and the listening device. Using the relative directions and the distance values, the primary device can generate a device map that enables the home theater to render output audio correctly.

Multi-device localization and ranging

A system configured to determine an estimated angle of arrival in reverberant environments. When a first device detects a calibration tone generated by a second device, the first device may generate multichannel audio representing the calibration tone and process the multichannel audio using a combination of detection filtering and subspace processing to determine a relative direction of the second device. For example, the first device may perform matched filtering to isolate a direct-path peak for the calibration tone, and then may sweep through all potential azimuth directions to identify an azimuth value corresponding to the direct-path peak. In some examples, the first device identifies a steering vector associated with a particular direction (e.g., signal subspace) that minimizes components in all other directions (e.g., noise subspace). The device may determine this steering vector independently for each frequency band and calculate the estimated angle of arrival by averaging results across frequency bands.

Direction of arrival estimation

A system configured to improve user localization used to determine a listening position and/or user orientation for a device map. Multiple devices may generate audio data representing user speech and the system may use the audio data to determine a first spatial likelihood function (SLF) based on angle measurements, determine a second SLF based on timing information, and determine a location of the user based on a combination of the two SLFs. The SLFs represent the environment using a grid comprising a plurality of grid cells, and each grid cell has a value indicating a likelihood that the grid cell corresponds to the location of the user. An individual device may generate a portion of the angle measurements based on multi-channel audio data generated using multiple microphones of the device, while the system may generate the timing information based on single-channel audio data received from each of the multiple devices.

User localization

A system that performs wall detection and localization to determine a position of a device relative to acoustically reflective surfaces. The device emits an audible sound including a frequency modulated signal and captures reflections of the audible sound. The frequency modulated signal enables the device to determine an amplitude of the reflections at different time-of-arrivals, which corresponds to a direction of the reflection. The device then performs beamforming to generate a 2D intensity map that represents an intensity of the reflections at each spatial location around the device. The device detects wall(s) in proximity to the device by identifying peak intensities represented in the 2D intensity map. In some examples, instead of performing beamforming, the device can perform directional wall detection by physically rotating the device and emitting the audible sound in multiple directions. The device may perform ultrasonic wall detection using ultrasonic sound frequencies.

Method for wall detection and localization

A system configured to create a flexible home theater group using a variety of different devices. To enable the home theater group to generate synchronized audio, the system performs device localization to generate map data, which represents locations of devices in a device map. The map data may include a listening position and/or television, such that the map data is centered on the listening position with the television along a vertical axis. To generate the map data, the system selects a primary device that determines calibration data indicating a sequence when each of the individual devices generates playback audio. The primary device sends the calibration data to secondary devices and each device generates playback audio at a designated time in the sequence, enabling other devices to capture the output audio and determine a relative position of the playback device (for example using angle of arrival and distance information).

Multi-device localization

A system configured to create a flexible home theater group using a variety of different devices. To enable synchronized audio output, the system performs device localization to generate map data representing locations of the devices. The system may generate the map data based on a listening position and/or location of a television, such that the map data is centered on the listening position with the television along a vertical axis. To generate the map data, the system processes measurement data representing relative directions between the devices and timing information that can be used to determine relative distances between the devices. Using the relative distances, the system determines relative positions of the devices (e.g., optimal arrangement). Knowing the relative directions and the relative positions, the system can determine an orientation of an individual device and use the orientation to determine the listening position and/or the location of the television.

Techniques for monitoring devices to use ultrasonic signals to detect and track the locations of moving objects in an environment. To determine distance information, the monitoring devices emit a frequency-modulated continuous wave (FMCW) signal at an ultrasound frequency range. Reflections of the FMCW ultrasonic signal are used to generate time-of-arrival (TOA) profiles that indicate distances between the monitoring device and objects in the environment. The reflections can be processed to suppress undesirable interferences, such as reflections off non-mobile objects in the environment (e.g., walls, furniture, etc.), vibrations off the floorings or the ceilings, etc. After processing the reflections, a heatmap can be used to plot the intensity of the reflections for the different TOAs of the reflections, and depict the movement of the user over time. Finally, a Kalman filter is used to smooth the peaks in the intensity values on the plot, and determine the trajectory of the human.

Monitoring non-stationary object distance using ultrasonic signals

Techniques for improving sound source localization (SSL) are provided. A method for probabilistic SSL using a deep neural network (DNN) may include receiving audio data including a representation of audio such as a wakeword from a microphone array. The audio data may be processed by a DNN to output a plurality of values where each value indicates a probability that the audio originated from a direction corresponding to that value. A sensor may provide computer vision or other data which may be used to inform the plurality of values based on detecting presence of a human or obstacle. A probability that the audio originated from one of the directions of the plurality of directions may be determined based at least in part on the DNN output and the computer vision or other data.

Sound source localization

Techniques for presence-detection devices to detect movement of a person in an environment by emitting ultrasonic signals using a loudspeaker that is concurrently outputting audible sound. To detect movement by the person, the devices characterize the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. However, when a loudspeaker plays audible sound while emitting the ultrasonic signal, audio signals generated by microphones of the devices include distortions caused by the loudspeaker. These distortions can be interpreted by the presence-detection devices as indicating movement of a person when there is no movement, or as indicating lack of movement when a user is moving. The techniques include processing audio signals to remove distortions to more accurately identify changes in the frequency of the reflections of the ultrasonic signals caused by the movement of the person.

Growing community of inventors

Jamaica Plain, MA, United States of America

Spencer Russell