Zoox, Inc.

Nam Gook Cho

Venkata Subrahmanyam Chandra Sekhar Chebiyyam

Subasingha Shaminda Subasingha

Jonathan Tyler Dowdall

Xuan Zhong

Subhasis Das

Mark Alan Bates

Shaminda Subasingha

Amanda Brown Prescott

Aleksandr Oysgelt

Sarah Tariq

Michael Carsten Bosse

Markus Jost

Turhan Karadeniz

Evan David Cook

Nathan Duarte

Dilip Bethanabhotla

David Donald Logan

Arvind Pattabhiraman

Marina Camille Josephs

Valerie Bumbaca Randolph

Caleb Heath Norfleet

MacKenzie Cunningham

Rohan Agrawal

Venkata Subrahrnanyarn Chandra Sekhar Chebiyyam

Systems and techniques for detecting and locating low impact collisions, such as relatively low energy impact collisions involving an autonomous vehicle, are disclosed herein. Sensor data from audio sensors configured on a vehicle may be used to determine whether a threshold number of such sensors have detected sounds associated with a collision. Scores determined for the sensors based on audio energy and confidence values can be used to determine a location estimate for the detected collision.

Event detection and localization using audio

An impact is detected with a vehicle having a body comprising parts which are deformable in an impact, by a secondary impact detection system configured to detect an impact that may not trigger a primary impact detection system. The secondary impact detection system comprises a plurality of deformation sensors attached to the deformable parts and configured to detect bending of the deformable parts. One or more processors are configured to receive a plurality of respective outputs from the plurality of deformation sensors, process the respective outputs to increase sensitivity to short-term changes relative to long-term changes and generate an indication of an impact dependent on detecting a processed output meeting a predetermined criterion. Outputs associated with a group of neighboring flex sensors may be used to localize the impact, characterize the source of impact, and/or reduce a likelihood of a false positive associated with the detection.

Detection of an impact with a vehicle

Techniques for cleaning and calibrating a microphone are discussed herein. For example, a computing device can implement a microphone component to operate a speaker adjacent to a microphone to cause removal of an obstruction (e.g., rain, mud, dirt, dust, snow, ice, animal droppings, etc.) on or near the microphone. The microphone component can also or instead cause the speaker to output a frequency for testing operation of the microphone. By implementing the cleaning and/or calibrating techniques described herein, foreign particle(s) of varying size and type can be dislodged from an area near the microphone to improve performance of the microphone.

Microphone cleaning and calibration

Techniques for determining information associated with sounds detected in an environment based on audio data and map data or perception data are discussed herein. A vehicle can use map data and/or perception data to distinguish between multiple audio signals or sounds. A direct source of sound can be distinguished from a reflected source of sound by determining a direction of arrival of sounds and which objects the directions of arrival are associated with in the environment. A reflected sound can be received without receiving a direct sound. Based on the reflected sound and map data or perception data, characteristics of sound in an occluded region of the environment may be determined and used to control the vehicle.

Distinguishing between direct sounds and reflected sounds in an environment

This disclosure relates to systems and techniques for identifying collisions, such as relatively low energy impact collisions involving an autonomous vehicle. Sensor data from a first sensor modality in a first array may be used to determine a first estimated location of impact and second sensor data from a second sensor modality in a second array may be used to determine a second estimated location of impact. A low energy impact event may be configured when the first estimated location of impact corresponds to the second estimated location of impact.

Low-energy impact collision detection

Techniques for utilizing microphone or audio data to detect and responding to low velocity impacts to a system such as an autonomous vehicle. In some cases, the system may be equipped with a plurality of microphones that may be used to detect impacts that fail to register on the data captured by the vehicle's inertial measurement units and may go undetected by the vehicle's perception system and sensors. In one specific example, the perception system of the autonomous vehicle may identify a period of time in which a potential low velocity impact may occur. The autonomous vehicle may then utilize the microphone or audio data associated with the period of time to determine if an impact occurred.

System to detect impacts

Detecting occluded objects using sound

Techniques for adaptive cross-correlation are discussed. A first signal is received from a first audio sensor associated with a vehicle and a second signal is received from a second audio sensor associated with the vehicle. Techniques may include determining, based at least in part on the first signal, a first transformed signal in a frequency domain. Additionally, the techniques include determining, based at least in part on the second signal, a second transformed signal in the frequency domain. A parameter can be determined based at least in part on a characteristic associated with at least one of the vehicle, an environment proximate the vehicle, or one or more of the first or second signal. Cross-correlation data can be determined based at least in part on one or more of the first transformed signal, the second transformed signal, or the parameter.

Adaptive cross-correlation

Techniques for determining information associated with sounds detected in an environment based on audio data are discussed herein. Audio sensors of a vehicle may determine audio data associated with sounds from the environment. Sounds may be caused by objects in the environment such as emergency vehicles, construction zones, non-emergency vehicles, humans, audio speakers, nature, etc. A model may determine a classification of the audio data and/or a probability value representing a likelihood that sound in the audio data is associated with the classification. A direction of arrival may be determined based on receiving classification values from multiple audio sensors of the vehicle, and other actions can be performed or the vehicle can be controlled based on the direction of arrival.

Emergency sound localization

Immersive experiences for users are described herein. In an example, audio data from a plurality of audio sensors associated with a vehicle can be received by an audio data processing system. The audio data processing system can combine individual captured audio channels (e.g., from the plurality of audio sensors) into two or more audio channels for output via two or more speakers proximate a user. A first audio channel of the two or more audio channels can be output via a first speaker and second audio channel of the two or more audio channels to be output via a second speaker, wherein output of the first audio channel and the second audio channel causes a resulting sound corresponding to at least a portion of a sound scene associated with the vehicle. In an example, a user computing device operable by the user can receive an input from the user.

Immersive sound for teleoperators

Techniques for estimating a direction of arrival of sound in an environment are discussed. First and second audio data are received from a first pair of audio sensors associated with a vehicle. A first region of ambiguity associated with the first pair of audio sensors is determined based on the first and second audio data. Third and fourth audio data are received from a second pair of audio sensors. A second region of ambiguity associated with the second pair of audio sensors is determined based on the third and fourth audio data. The regions of ambiguity can be further based on confidence levels associated with sensor or audio data. An area of intersection of the first region of ambiguity and the second region of ambiguity can be determined. A direction of arrival of an audio event can be determined based on the area of intersection.

Direction of arrival estimation

Techniques for determining a direction of arrival of an emergency are discussed. A plurality of audio sensors of a vehicle can receive audio data associated with the vehicle. An audio sensor pair can be selected from the plurality of audio sensors to generate audio data representing sound in an environment of the vehicle. An angular spectrum associated with the audio sensor pair can be determined based on the audio data. A feature associated with the audio data can be determined based on the angular spectrum and/or the audio data itself. A direction of arrival (DoA) value associated with the sound may be determined based on the feature using a machine learned model. An emergency sound (e.g., a siren) can be detected in the audio data and a direction associated with the emergency relative to the vehicle can be determined based on the feature and the DoA value.

Growing community of inventors

Cupertino, CA, United States of America

Nam Gook Cho