Zoox, Inc.

Kratarth Goel

Sarah Tariq

James William Vaisey Philbin

Vasiliy Karasev

Praveen Srinivasan

Adam Pollack

Yi Xu

Xin Wang

Oytun Ulutan

Jesse Sol Levinson

Derek Xiang Ma

Scott M Purdy

Jonathan Tyler Dowdall

Ekaterina Hristova Taralova

Justin Nordgreen

Thomas Oscar Dudzik

Bharadwaj Raghavan

Techniques are discussed herein for generating three-dimensional (3D) representations of an environment based on two-dimensional (2D) image data, and using the 3D representations to perform 3D object detection and other 3D analyses of the environment. 2D image data may be received, along with depth estimation data associated with the 2D image data. Using the 2D image data and associated depth data, an image-based object detector may generate 3D representations, including point clouds and/or 3D pixel grids, for the 2D image or particular regions of interest. In some examples, a 3D point cloud may be generated by projecting pixels from the 2D image into 3D space followed by a trained 3D convolutional neural network (CNN) performing object detection. Additionally or alternatively, a top-down view of a 3D pixel grid representation may be used to perform object detection using 2D convolutions.

Three-dimensional point clouds based on images and depth data

Techniques for adjusting vehicle models based on environmental conditions are discussed herein. The techniques may include receiving image data representing a portion of an environment in which a vehicle is operating and inputting the image data into a machine learned model. Additionally, data representing an environmental condition associated with the environment may be received from a sensor of the vehicle to detect changes in the environmental conditions such that one or more actions associated with the machine learned model or an output of the machine learned model may be performed. Some of the techniques may also include running multiple machine learned models or multiple configurations of a machine learned model in parallel and selecting different outputs of the machine learned model(s) to adjust for changes in the environmental conditions. For instance, individual outputs may be selected based on environmental conditions, confidence scores, thresholds, etc.

Adjusting vehicle models based on environmental conditions

Techniques for refining an image-based bounding box with depth data from a lidar sensor to represent an object in an environment of a vehicle are discussed herein. A vehicle computing device can implement a model to determine a three-dimensional bounding box based on image data. The vehicle computing device can implement a same of different model to update the three-dimensional bounding box based at least in part on depth information associated with lidar data. The techniques can also include training one or more machine learned models to determine three-dimensional bounding boxes usable by a vehicle to navigate in the environment.

Vision based object representation

Parallel vehicle model processing

Techniques for detecting attributes and/or gestures associated with pedestrians in an environment are described herein. The techniques may include receiving sensor data associated with a pedestrian in an environment of a vehicle and inputting the sensor data into a machine-learned model that is configured to determine a gesture and/or an attribute of the pedestrian. Based on the input data, an output may be received from the machine-learned model that indicates the gesture and/or the attribute of the pedestrian and the vehicle may be controlled based at least in part on the gesture and/or the attribute of the pedestrian. The techniques may also include training the machine-learned model to detect the attribute and/or the gesture of the pedestrian.

Pedestrian attribute and gesture detection

Techniques for detecting key points associated with objects in an environment are described herein. The techniques may include receiving sensor data representing a portion of an environment in which the vehicle is operating and inputting the sensor data into a machine-learned model. Based on the input sensor data, the machine-learned model may detect one or more key points corresponding to physical features (e.g., hands, feet, eyes, etc.) of a pedestrian who is in the environment. Based on the one or more key points, a bounding box associated with the pedestrian may be generated and the vehicle may be controlled based on at least one of the key points or the bounding box. The techniques may also include training the machine-learned model to detect key points associated with pedestrians.

Key point detection

Techniques for utilizing a depth completion algorithm to determine dense depth data are discussed are discussed herein. Two-dimensional image data representing an environment can be captured or otherwise received. Depth data representing the environment can be captured or otherwise received. The depth data can be projected into the image data and processed using the depth completion algorithm. The depth completion algorithm can be utilized to determine the dense depth values based on intensity values of pixels, and other depth values. A vehicle can be controlled based on the determined depth values.

Associating image data and depth data for depth determination

Three-dimensional object detection based on image data

Machine-learned model training for pedestrian attribute and gesture detection

A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.

Multi-task learning for real-time semantic and/or depth aware instance segmentation and/or three-dimensional object bounding

Techniques for training a machine learned (ML) model to determine depth data based on image data are discussed herein. Training can use stereo image data and depth data (e.g., lidar data). A first (e.g., left) image can be input to a ML model, which can output predicted disparity and/or depth data. The predicted disparity data can be used with second image data (e.g., a right image) to reconstruct the first image. Differences between the first and reconstructed images can be used to determine a loss. Losses may include pixel, smoothing, structural similarity, and/or consistency losses. Further, differences between the depth data and the predicted depth data and/or differences between the predicted disparity data and the predicted depth data can be determined, and the ML model can be trained based on the various losses. Thus, the techniques can use self-supervised training and supervised training to train a ML model.

Depth data model training with upsampling, losses and loss balancing

Using detection of low variance regions for improving detection is described. In an example, sensor data can be received from a sensor associated with a vehicle. The sensor data can represent an environment. An indication of a low variance region associated with the sensor data can be determined and an indication of a high variance region associated with the sensor data can be determined based at least in part on the indication of the low variance region. The vehicle can be controlled based on at least one of the sensor data or the indication of the high variance region.

Low variance region detection for improved high variance region detection using machine learning

Low variance detection training is described herein. In an example, annotated data can be determined based on sensor data received from a sensor associated with a vehicle. The annotated data can comprise an annotated low variance region and/or an annotated high variance region. The sensor data can be input into a model, and the model can determine an output comprising a high variance output and a low variance output. In an example, a difference between the annotated data and the output can be determined and one or more parameters associated with the model can be altered based at least in part on the difference. The model can be transmitted to a vehicle configured to be controlled by another output of the model.

Training a machine-learned model to detect low variance regions

Techniques for utilizing multiple scales of images as input to machine learning (ML) models are discussed herein. Operations can include providing an image associated with a first scale to a first ML model. An output of the first ML model can include a first bounding box indicative of a first region of the image representing a first object, with the first bounding box falling within a first range of sizes. Next, a scaled image can be generated by scaling the image. The scaled image can be provided to a second ML model, which can output a second bounding box indicative of a second region of the image representing a second object, the second bounding falling within a second range of sizes. Thus, inputting a scaled image to a same ML model (or to different ML models) can result in different detected features in the images.

Growing community of inventors

Albany, CA, United States of America

Kratarth Goel