Amazon Technologies, Inc.

Krishna Kamath Koteshwara

Trausti Thor Kristjansson

Tarun Pruthi

Spencer Russell

Harsha Inna Kedage Rao

Rui Wang

Yuzhou Liu

Anran Wang

Robert Ayrapetian

Ke Li

Mashhour Solh

Ali Abdollahzadeh Milani

Pushkaraksha Gejji

Wan-Chieh Pai

Navin Chatlani

Maziyar Khorasani

Zhen Sun

Vamshi Krishna Chillara

Inseok Heo

Techniques for user devices to coordinate the emission of ultrasonic signals on behalf of different ultrasonic applications running on the user devices. User devices may have an ultrasonic front end that provides various ultrasonic applications, such as presence detection and proximity sensing, sleep monitoring, temperature and humidity sensing, and heartbeat monitoring. To perform the different functions, the ultrasonic applications use active sensing, or the transmission of ultrasonic signals to collect data about the environments around the user devices. However, the ultrasonic applications often use specific ultrasonic signals having different waveforms in order to collect the data used for their specific functionality. The ultrasonic front end may perform techniques to coordinate mutually exclusive emissions of the ultrasonic signals on behalf of the different ultrasonic applications based on various considerations, such as application priorities, operating schedules of the applications, and applications that can use the same waveform to collect data.

Coordinating presence detection and proximity sensing using ultrasonic signals

A system that performs wall detection and localization to determine a position of a device relative to acoustically reflective surfaces. The device emits an audible sound including a frequency modulated signal and captures reflections of the audible sound. The frequency modulated signal enables the device to determine an amplitude of the reflections at different time-of-arrivals, which corresponds to a direction of the reflection. The device then performs beamforming to generate a 2D intensity map that represents an intensity of the reflections at each spatial location around the device. The device detects wall(s) in proximity to the device by identifying peak intensities represented in the 2D intensity map. In some examples, instead of performing beamforming, the device can perform directional wall detection by physically rotating the device and emitting the audible sound in multiple directions. The device may perform ultrasonic wall detection using ultrasonic sound frequencies.

Method for wall detection and localization

Techniques for presence-detection devices to vary presence-detection sensitivity to detect different types of movements by objects, such as major movements and minor movements, using ultrasonic signals. The devices detect movement of a person in an environment by emitting ultrasonic signals into the environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person relative to the presence-detection devices. The presence-detection devices may control the presence-detection sensitivity in order to detect major movements, such as a user walking in a room, as well as objects minor movements, such as a user typing at a computer. By adjusting the presence-detection sensitivity, the devices are able to improve the overall accuracy of detecting the presence of users in an environment.

Controlling sensitivity of presence detection using ultrasonic signals

Techniques for monitoring devices to use ultrasonic signals to detect and track the locations of moving objects in an environment. To determine distance information, the monitoring devices emit a frequency-modulated continuous wave (FMCW) signal at an ultrasound frequency range. Reflections of the FMCW ultrasonic signal are used to generate time-of-arrival (TOA) profiles that indicate distances between the monitoring device and objects in the environment. The reflections can be processed to suppress undesirable interferences, such as reflections off non-mobile objects in the environment (e.g., walls, furniture, etc.), vibrations off the floorings or the ceilings, etc. After processing the reflections, a heatmap can be used to plot the intensity of the reflections for the different TOAs of the reflections, and depict the movement of the user over time. Finally, a Kalman filter is used to smooth the peaks in the intensity values on the plot, and determine the trajectory of the human.

Monitoring non-stationary object distance using ultrasonic signals

Techniques for presence-detection devices to emission levels of ultrasonic signals that are used to detect movement in an environment. The presence-detection devices may detect movement of a person by emitting the ultrasonic signals into an environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person relative to the presence-detection devices. However, presence-detection devices that continuously emit ultrasonic signals may experience reduced battery life, increased likelihood of overheating, etc. To reduce these negative effects, the presence-detection devices may reduce the emission levels of ultrasonic signals. For instance, once motion is detected, the presence-detection devices may, for a period of time, stop emitting ultrasonic signals or reduce the power level at which the ultrasonic signals are emitted. Accordingly, the presence-detection devices can reduce emission levels of ultrasonic signals while still detecting motion.

Controlling emission of ultrasonic signals for presence detection

Techniques for presence-detection devices to detect movement of a person in an environment by emitting ultrasonic signals using a loudspeaker that is concurrently outputting audible sound. To detect movement by the person, the devices characterize the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. However, when a loudspeaker plays audible sound while emitting the ultrasonic signal, audio signals generated by microphones of the devices include distortions caused by the loudspeaker. These distortions can be interpreted by the presence-detection devices as indicating movement of a person when there is no movement, or as indicating lack of movement when a user is moving. The techniques include processing audio signals to remove distortions to more accurately identify changes in the frequency of the reflections of the ultrasonic signals caused by the movement of the person.

Presence detection using ultrasonic signals with concurrent audio playback

Techniques for calibrating presence-detection devices to account for various factors that can affect the presence-detection devices' ability to detect movement. Presence-detection devices may detect movement of a person in an environment by emitting ultrasonic signals into the environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. However, factors such as environmental acoustic conditions, noise sources, etc., may affect the ability of the presence-detection devices to detect movement. To calibrate for these factors, the presence-detection devices may use a loudspeaker to emit an ultrasonic sweep signal that spans different frequencies in an ultrasonic frequency range. The presence-detection devices may generate audio data using a microphone that represents the ultrasonic sweep signal, and analyze that audio data to determine an optimal frequency range and/or transmission power for subsequent ultrasonic signal transmissions.

Device calibration for presence detection using ultrasonic signals

This disclosure describes presence-detection devices that detect movement of a person by emitting ultrasonic signals into an environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. The techniques include downsampling the audio signals from the carrier frequency range down to a frequency range with a center frequency around 0 Hz. A filter is applied to attenuate signals around 0 Hz and below (or above), such as the emitted signals. In addition to removing the emitted signals, the negative side (or positive side) of the audio signals are removed, but the Doppler shift is still represented in the remaining portion of the audio signals. By removing a portion of the audio signals, the amount of processing required to detect the Doppler shift in the reflections of the ultrasonic signals is reduced.

Processing audio signals for presence detection

A system configured to improve echo cancellation for nonlinear systems. The system generate reference audio data by isolating portions of microphone audio data that correspond to playback audio data. For example, the system may determine a correlation between the playback audio data and the microphone audio data in individual time-frequency bands in a frequency domain. In some examples, the system may substitute microphone audio data associated with output audio for the playback audio data. The system may generate the reference audio data based on portions of the microphone audio data that have a strong correlation with the playback audio data. The system may generate the reference audio data by selecting these portions of the microphone audio data or by performing beamforming. This results in precise time alignment between the reference audio data and the microphone audio data, improving performance of the echo cancellation.

Echo cancellation by acoustic playback estimation

This disclosure describes presence-detection devices that detect movement of a person in an environment by emitting ultrasonic signals into the environment, and characterizing the change in the frequency, or the Doppler shift, of the reflections of the ultrasonic signals off the person caused by the movement of the person. In addition to detecting movement, and thus presence of a person, the presence-detection devices may include a microphone array to perform techniques for identifying a direction of movement of the person, and also to perform techniques for identifying a number of people that are in the room. Additionally, the techniques described herein include processing audio signals in such a way to allow for the use of on-board loudspeakers to transmit ultrasonic signals at out-of-band frequencies.

Presence detection using ultrasonic signals

A system configured to reconstruct audio signals. The system may identify missing audio samples due to packet loss or detect distortion caused by audio clipping and may reconstruct the audio data. The system may employ a forward-looking neural network that recursively predicts audio samples based on previous audio samples and/or a backward-looking neural network that recursively predicts audio samples based on subsequent audio samples. The system may generate audio data using only the forward-looking neural network for low latency applications or may generate audio data using both neural networks for mid to high latency applications. To reduce distortion in output audio data, the system may generate the audio data by cross-fading between outputs of the neural networks and/or may cross-fade between the generated audio data and the input audio data.

Methods for reconstructing an audio signal

A device that determines an echo latency estimate by combining reference signals. The device may determine the echo latency corresponding to an amount of time between reference signals being sent to transmitters and input data corresponding to the reference signals being received. The device may generate a combined reference signal by adding (or filtering) each of the reference signals. The device may then compare the combined reference signal to input audio data received from a microphone or receiving device. The device may detect a highest peak, determine if there are any earlier significant peaks and estimate the echo latency based on the earliest significant peak. This technique is not limited to audio data and may be used for signal matching using any system that includes multiple transmitters and receivers (e.g., Radar, Sonar, etc.).

Echo latency estimation

A speech recognition computer system uses video input as well as audio input of known speech when the speech recognition computer system is being trained to recognize unknown speech. The video of the speaker can be captured using multiple cameras, from multiple angles. The audio can be captured using multiple microphones. The video and audio can be sampled so that timing of events in the video and audio can be determined from the content independent of an audio or video capture device's clock. Video features, such as a speaker's moving body parts, can be extracted from the video and random sampled, to be used in a speech modeling process. Audio is modeled at the phoneme level, which provides word mapping with minor additional effort. The trained speech recognition computer system can then be used to recognize speech text from video/audio of unknown speech.

On-the-fly speech learning and computer model generation using audio-visual synchronization

An ultrasonic finger print authentication system that generates a three-dimensional representation of a fingerprint. High frequency sound waves are used as a medium for imaging finer details of the patterns on a fingertip, including the ridge and valley formations. Multiple frequencies and beam-forming are used to quickly map the ridges. Acoustic resonance is used to determine the depths of the intervening valleys.

Growing community of inventors

Santa Clara, CA, United States of America

Krishna Kamath Koteshwara