Zoox, Inc.

Praveen Srinivasan

Sarah Tariq

Kratarth Goel

James William Vaisey Philbin

Thomas Oscar Dudzik

A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.

Multi-task learning for real-time semantic and/or depth aware instance segmentation and/or three-dimensional object bounding

A vehicle can use an image sensor to both detect objects and determine depth data associated with the environment the vehicle is traversing. The vehicle can capture image data and lidar data using the various sensors. The image data can be provided to a machine-learned model trained to output depth data of an environment. Such models may be trained, for example, by using lidar data and/or three-dimensional map data associated with a region in which training images and/or lidar data were captured as ground truth data. The autonomous vehicle can further process the depth data and generate additional data including localization data, three-dimensional bounding boxes, and relative depth data and use the depth data and/or the additional data to autonomously traverse the environment, provide calibration/validation for vehicle sensors, and the like.

Image-based depth data and localization

Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Depth data model training with upsampling, losses and loss balancing

Depth estimates for an object made by one or more sensors of a vehicle may be refined using locations of environmental attributes that are proximate the object. An image captured of the object proximate an environmental attribute may be analyzed to determine where the object is positioned relative to the environmental attribute. A machine-learned model may be used to detect the environmental attribute, and a location of the environmental attribute may be determined from map data. A probability of a location of the object may be determined based on the known location of the environmental attribute. The location probability of the object may be used to refine depth estimates generated by other means, such as a monocular depth estimation from an image using computer vision.

Refining depth from an image

Depth data model training with upsampling, losses, and loss balancing

Image-based depth data and relative depth data

Training a machine-learning (ML) architecture to determine three or more outputs at a rate of 30 or more frames per second on consumer grade hardware may comprise jointly training components of the ML using loss(es) determined across the components and/or consistency losses determined between outputs of two or more components. The ML architecture discussed herein may comprise one or more sets of neural network layers and/or respective components for determining a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image.

Growing community of inventors

San Francisco, CA, United States of America

Praveen Srinivasan