Zoox, Inc.

Scott M Purdy

Kratarth Goel

Jonathan Tyler Dowdall

Adam Pollack

Bharadwaj Raghavan

Subhasis Das

Zeng Wang

Derek Xiang Ma

Clement Besson

Tonghan Gu

Techniques for determining occupancy using unobstructed sensor emissions. For instance, a vehicle may receive sensor data from one or more sensors. The sensor data may represent at least locations to points within an environment. Using the sensor data, the vehicle may determine areas within the environment that are obstructed by objects (e.g., locations where objects are located). The vehicle may also use the sensor data to determine areas within the environment that are unobstructed by objects (e.g., locations where objects are not located). In some examples, the vehicle determines the unobstructed areas as including areas that are between the vehicle and the identified objects. This is because sensor emissions from the sensor(s) passed through these areas and then reflected off of objects located farther distances from the vehicle. The vehicle may then generate a map indicating at least the obstructed areas and the unobstructed areas within the environment.

Determining occupancy using unobstructed sensor emissions

Techniques are discussed herein for controlling autonomous vehicles within a driving environment, including generating and using bounding contours associated with objects detected in the environment. Image data may be captured and analyzed to identify and/or classify objects within the environment. Image-based and/or lidar-based techniques may be used to determine depth data associated with the objects, and a bounding contour may be determined based on the object boundaries and associated depth data. An autonomous vehicle may use the bounding contours of objects within the environment to classify the objects, predict the positions, poses, and trajectories of the objects, and determine trajectories and perform other vehicle control actions while safely navigating the environment.

Object bounding contours based on image data

Techniques for determining attributes associated with objects represented in temporal sensor data. In some examples, the techniques may include receiving sensor data including a representation of an object in an environment. The sensor data may be generated by a temporal sensor of a vehicle and, in some instances, a trajectory of the object or the vehicle may contribute to a distortion in the representation of the object. For instance, a shape of the representation of the object may be distorted relative to an actual shape of the object. The techniques may also include determining an attribute (e.g., velocity, bounding box, etc.) associated with the object based at least in part on a difference between the representation of the object and another representation of the object (e.g., in other sensor data) or the actual shape of the object.

Inferring object attributes from track associations

Techniques for refining an image-based bounding box with depth data from a lidar sensor to represent an object in an environment of a vehicle are discussed herein. A vehicle computing device can implement a model to determine a three-dimensional bounding box based on image data. The vehicle computing device can implement a same of different model to update the three-dimensional bounding box based at least in part on depth information associated with lidar data. The techniques can also include training one or more machine learned models to determine three-dimensional bounding boxes usable by a vehicle to navigate in the environment.

Vision based object representation

Techniques for reducing latency associated with temporal sensor data processing. The techniques may include determining a region of interest within a field of view of a temporal sensor (e.g., rotating lidar sensor, rolling shutter camera, etc.). Based at least in part on determining the region of interest, an azimuth angle associated with a partial scan of the temporal sensor may be determined such that the region of interest is disposed within the azimuth angle. The techniques may also include receiving sensor data from the temporal sensor and determining whether the sensor data corresponds with the azimuth angle/region of interest. Based at least in part on a determination that the sensor data corresponding with the azimuth angle/region of interest has been received, the sensor data may be processed prior to at least one of receiving additional sensor data or a completion of a full scan by the temporal sensor.

Partial sensor data processing for latency reduction

Machine-learned model training for inferring object attributes from distortions in temporal data

Inferring object attributes from distortions in temporal data

Object tracking based on temporal data attribute inferences

Determining object characteristics using unobstructed sensor emissions

Techniques for evaluating and validating progress of training machine-learned models are described herein. The techniques may include storing, in a database, metric data associated with outputs from machine-learned models based on sensor data inputs. For instance, the metric data may include first metric data associated with a first bounding box output by a machine-learned model and second metric data associated with a second bounding box output by an updated version of the machine-learned model. The techniques also include a graphical user interface (GUI) for presenting visualizations of the metric data that improve the ability to evaluate the performance of a machine-learned model. In some examples, an indication of a request to evaluate the updated version of the machine-learned model may be received via the GUI. Based on the indication, the GUI may cause presentation of visualization(s) of difference(s) between first metric(s) of the first metric data and second metric(s) of the second metric data.

Metric visualization system for model evaluation

Techniques for utilizing a depth completion algorithm to determine dense depth data are discussed are discussed herein. Two-dimensional image data representing an environment can be captured or otherwise received. Depth data representing the environment can be captured or otherwise received. The depth data can be projected into the image data and processed using the depth completion algorithm. The depth completion algorithm can be utilized to determine the dense depth values based on intensity values of pixels, and other depth values. A vehicle can be controlled based on the determined depth values.

Growing community of inventors

Lake Forest Park, WA, United States of America

Scott M Purdy