Zoox, Inc.

Tencia Lee

James William Vaisey Philbin

Kai Zhenyu Wang

Dragomir Dimitrov Anguelov

Sabeek Mani Pradhan

Sarah Tariq

Vasiliy Karasev

Matthew Van Heukelom

A monocular image often does not contain enough information to determine, with certainty, the depth of an object in a scene reflected in the image. Combining image data and LIDAR data may enable determining a depth estimate of the object relative to the camera. Specifically, LIDAR points corresponding to a region of interest ('ROI') in the image that corresponds to the object may be combined with the image data. These LIDAR points may be scored according to a monocular image model and/or a factor based on a distance between projections of the LIDAR points into the ROI and a center of the region of interest may improve the accuracy of the depth estimate. Using these scores as weights in a weighted median of the LIDAR points may improve the accuracy of the depth estimate, for example, by discerning between a detected object and an occluding object and/or background.

Associating LIDAR data and image data

Techniques for determining an uncertainty metric associated with an object in an environment can include determining the object in the environment and a set of candidate trajectories associated with the object. Further, a vehicle, such as an autonomous vehicle, can be controlled based at least in part on the uncertainty metric. The vehicle can determine a traversed trajectory associated with the object and determine a difference between the traversed trajectory and the set of candidate trajectories. Based on the difference, the vehicle can determine an uncertainty metric associated with the object. In some instances, the vehicle can input the traversed trajectory and the set of candidate trajectories to a machine-learned model that can output the uncertainty metric.

Object uncertainty detection

Systems and methods for estimating a height of an object from a monocular image are described herein. Objects are detected in the image, each object being indicated by a region of interest. The image is then cropped for each region of interest and the cropped image scaled to a predetermined size. The cropped and scaled image is then input into a convolutional neural network (CNN), the output of which is an estimated height for the object. The height may be represented by a mean of a probability distribution of possible sizes, a standard deviation, as well as a level of confidence. A location of the object may be determined based on the estimated height and region of interest. A ground truth dataset may be generated for training the CNN by simultaneously capturing a LIDAR sequence with a monocular image sequence.

Object height estimation from monocular images

Techniques for determining and/or predicting a trajectory of an object by using the appearance of the object, as captured in an image, are discussed herein. Image data, sensor data, and/or a predicted trajectory of the object (e.g., a pedestrian, animal, and the like) may be used to train a machine learning model that can subsequently be provided to, and used by, an autonomous vehicle for operation and navigation. In some implementations, predicted trajectories may be compared to actual trajectories and such comparisons are used as training data for machine learning.

Motion prediction based on appearance

Techniques for determining lighting states of a tracked object, such as a vehicle, are discussed herein. An autonomous vehicle can include an image sensor to capture image data of an environment. Objects such can be identified in the image data as objects to be tracked. Frames of the image data representing the tracked object can be selected and input to a machine learning algorithm (e.g., a convolutional neural network, a recurrent neural network, etc.) that is trained to determine probabilities associated with one or more lighting states of the tracked object. Such lighting states include, but are not limited to, a blinker state(s), a brake state, a hazard state, etc. Based at least in part on the one or more probabilities associated with the one or more lighting states, the autonomous vehicle can determine a trajectory for the autonomous vehicle and/or can determine a predicted trajectory for the tracked object.

Growing community of inventors

San Francisco, CA, United States of America

Tencia Lee