Zoox, Inc.

Jonathan Tyler Dowdall

Venkata Subrahmanyam Chandra Sekhar Chebiyyam

Subasingha Shaminda Subasingha

Nam Gook Cho

Kratarth Goel

Scott M Purdy

Adam Pollack

Sarah Tariq

Michael Carsten Bosse

Derek Xiang Ma

Amanda Brown Prescott

Bharadwaj Raghavan

Amir Ghaderi

Dilip Bethanabhotla

Techniques for refining an image-based bounding box with depth data from a lidar sensor to represent an object in an environment of a vehicle are discussed herein. A vehicle computing device can implement a model to determine a three-dimensional bounding box based on image data. The vehicle computing device can implement a same of different model to update the three-dimensional bounding box based at least in part on depth information associated with lidar data. The techniques can also include training one or more machine learned models to determine three-dimensional bounding boxes usable by a vehicle to navigate in the environment.

Vision based object representation

Techniques for detecting, locating, and/or classifying objects based on multiple sensor data inputs received from different sensor modalities. The techniques may include receiving sensor data generated by different sensor modalities of a vehicle, the sensor data including at least first sensor data generated by a first sensor modality and second sensor data generated by a second sensor modality. In some examples, the sensor data may be input into a machine-learning pipeline. The machine-learning pipeline may be configured to determine locations of objects in an environment surrounding the vehicle based at least in part on a correlation, by the multi-attention component, of the first sensor data and the second sensor data. The techniques may also include receiving, from the machine-learning pipeline, an output indicating a location of an object in the environment.

Multi-attention machine learning for object detection and classification

Techniques for utilizing a depth completion algorithm to determine dense depth data are discussed are discussed herein. Two-dimensional image data representing an environment can be captured or otherwise received. Depth data representing the environment can be captured or otherwise received. The depth data can be projected into the image data and processed using the depth completion algorithm. The depth completion algorithm can be utilized to determine the dense depth values based on intensity values of pixels, and other depth values. A vehicle can be controlled based on the determined depth values.

Associating image data and depth data for depth determination

Techniques for utilizing microphone or audio data to detect and responding to low velocity impacts to a system such as an autonomous vehicle. In some cases, the system may be equipped with a plurality of microphones that may be used to detect impacts that fail to register on the data captured by the vehicle's inertial measurement units and may go undetected by the vehicle's perception system and sensors. In one specific example, the perception system of the autonomous vehicle may identify a period of time in which a potential low velocity impact may occur. The autonomous vehicle may then utilize the microphone or audio data associated with the period of time to determine if an impact occurred.

System to detect impacts

Techniques for determining information associated with sounds detected in an environment based on audio data are discussed herein. Audio sensors of a vehicle may determine audio data associated with sounds from the environment. Sounds may be caused by objects in the environment such as emergency vehicles, construction zones, non-emergency vehicles, humans, audio speakers, nature, etc. A model may determine a classification of the audio data and/or a probability value representing a likelihood that sound in the audio data is associated with the classification. A direction of arrival may be determined based on receiving classification values from multiple audio sensors of the vehicle, and other actions can be performed or the vehicle can be controlled based on the direction of arrival.

Emergency sound localization

Immersive experiences for users are described herein. In an example, audio data from a plurality of audio sensors associated with a vehicle can be received by an audio data processing system. The audio data processing system can combine individual captured audio channels (e.g., from the plurality of audio sensors) into two or more audio channels for output via two or more speakers proximate a user. A first audio channel of the two or more audio channels can be output via a first speaker and second audio channel of the two or more audio channels to be output via a second speaker, wherein output of the first audio channel and the second audio channel causes a resulting sound corresponding to at least a portion of a sound scene associated with the vehicle. In an example, a user computing device operable by the user can receive an input from the user.

Immersive sound for teleoperators

Techniques for determining a direction of arrival of an emergency are discussed. A plurality of audio sensors of a vehicle can receive audio data associated with the vehicle. An audio sensor pair can be selected from the plurality of audio sensors to generate audio data representing sound in an environment of the vehicle. An angular spectrum associated with the audio sensor pair can be determined based on the audio data. A feature associated with the audio data can be determined based on the angular spectrum and/or the audio data itself. A direction of arrival (DoA) value associated with the sound may be determined based on the feature using a machine learned model. An emergency sound (e.g., a siren) can be detected in the audio data and a direction associated with the emergency relative to the vehicle can be determined based on the feature and the DoA value.

Growing community of inventors

San Francisco, CA, United States of America

Jonathan Tyler Dowdall