Zoox, Inc.

Sabeek Mani Pradhan

Cooper Stokes Sloan

Dragomir Dimitrov Anguelov

Subhasis Das

Chuang Wang

Joshua Kriser Cohen

Tencia Lee

Balazs Kovacs

Techniques for updating data operations in a perception system are discussed herein. A vehicle may use a perception system to capture data about an environment proximate to the vehicle. The perception system may receive image data, lidar data, and/or radar data to determine information about an object in the environment. As different sensors may be associated with different time periods for capturing and/or processing operations, the techniques include updating object data with data from sensors associated with a shorter time period to generate intermediate object data. Such intermediate object data may reduce a delay in updating a position of an object in an environment.

Out of sequence perception system

Techniques for updating data operations in a perception system are discussed herein. A vehicle may use a perception system to capture data about an environment proximate to the vehicle. The perception system may receive image data, lidar data, and/or radar data to determine information about an object in the environment. As different sensors may be associated with different time periods for capturing and/or processing operations, the techniques include updating object data with data from sensors associated with a shorter time period to generate intermediate object data.

Out of sequence preception system

A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest ('ROI') in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

Associating LIDAR data and image data

A machine-learning architecture may be trained to determine point cloud data associated with different types of sensors with an object detected in an image and/or generate a three-dimensional region of interest (ROI) associated with the object. In some examples, the point cloud data may be associated with sensors such as, for example, a lidar device, radar device, etc.

Multi-modal sensor data association architecture

Ground truth data may be too sparse to supervise training of a machine-learned (ML) model enough to achieve an ML model with sufficient accuracy/recall. For example, in some cases, ground truth data may only be available for every third, tenth, or hundredth frame of raw data. Training an ML model to detect a velocity of an object when ground truth data for training is sparse may comprise training the ML model to predict a future position of the object based at least in part on image, radar, and/or lidar data (e.g., for which no ground truth may be available). The ML model may be altered based at least in part on a difference between ground truth data associated with a future time and the future position.

Growing community of inventors

Redwood City, CA, United States of America

Sabeek Mani Pradhan