Zoox, Inc.

Zeng Wang

David Pfeiffer

Subhasis Das

Dragomir Dimitrov Anguelov

James William Vaisey Philbin

Bertrand Robert Douillard

Robert Evan Mahieu

Yuanyuan Chen

Allan Zelener

Jesse Sol Levinson

Nitesh Shroff

Janek Hudecek

Derek Xiang Ma

Scott M Purdy

Arthur Daniel Costea

Brice Rebsamen

Mark Jonathon McClelland

Carden Taylor Bagwell

Dhanushka Nirmevan Kularatne

Qiang Zhai

Troy Donovan O′Neal

Techniques are discussed herein for controlling autonomous vehicles within a driving environment, including generating and using bounding contours associated with objects detected in the environment. Image data may be captured and analyzed to identify and/or classify objects within the environment. Image-based and/or lidar-based techniques may be used to determine depth data associated with the objects, and a bounding contour may be determined based on the object boundaries and associated depth data. An autonomous vehicle may use the bounding contours of objects within the environment to classify the objects, predict the positions, poses, and trajectories of the objects, and determine trajectories and perform other vehicle control actions while safely navigating the environment.

Object bounding contours based on image data

Techniques for determining a probability that an object that has been appropriately labeled as a non-impeding object are disclosed. Non-impeding objects may be objects in an environment that do not substantially affect the operation of a vehicle in the environment, such as steam, fog, smoke, etc. A machine-learned model may classify such objects as non-impeding objects. The designation may be checked by using the lidar data and measurements associated with the lidar points associated with the object to determine a probability that the object is a non-impeding object. The lidar data may be aggregated within the individual voxels of a voxelized representation of the object for use in the probability determination.

Object detection soft consistency checker

Techniques for densifying annotations associated with sensor data are discussed herein. For example, techniques may include projecting a subset of lidar points into an unannotated two-dimensional image. Based on the projecting, the annotations of lidar points projected into the image can be associated with the image data. A subset of pixels of pixels can be dilated, and the dilated pixels can be determined to intersect (e.g., overlap, touch, etc.). Based on the pixels intersecting (or being adjacent) and being associated with the same segment identifier, a contour can be rendered around the corresponding non-dilated pixels to identify an object in the image data.

Annotating image data based on projected lidar data

Techniques for identifying lidar points associated with static objects, and using such lidar points to annotate objects within two-dimensional images are discussed herein. In some examples, an object manager may receive accumulations of lidar data captured from lidar devices of a vehicle while traversing within a driving environment. In some examples, the object manager may receive a plurality of annotated images. Such annotations may identify static objects within the driving environment. In some instances, the object manager may project a lidar point into an annotated image and determine that the lidar point is associated with an annotated pixel. Based on the pixel being associated with the annotated object, the object manager may determine that the lidar point is associated with object. In some examples, the object manager may determine a subset of lidar points that are associated with the object.

Lidar debris detection based on annotated image data

Techniques for detecting and classifying objects using lidar data are discussed herein. In some cases, the system may be configured to utilize a predetermined number of prior frames of lidar data to assist with detecting and classifying objects. In some implementations, the system may utilize a subset of the data associated with the prior lidar frames together with the full set of data associated with a current frame to detect and classify the objects.

Method for predicting behavior of object e as an autonomous vehicles, determining afggregated data based on first and second lidars data and transfer function

A vehicle can include various sensors to detect objects in an environment. In some cases, the object may be within a planned path of travel of the vehicle. In these cases, leaving the planned path may be dangerous to the passengers so the vehicle may, based on dimensions of the object, dimensions of the vehicle, and semantic information of the object, determine operational parameters associate with passing the object while maintaining a position within the planned path, if possible.

In-path obstacle detection and avoidance system

Techniques for determining ground surface voxels based on multiresolution planes are discussed herein. Such multiresolution planes may be generated, validated, and used to connect voxels associated with the ground. In some examples, a vehicle may use capture or otherwise receive lidar data while traversing within a driving environment. The vehicle may associate the lidar data with a voxel space. Voxels can be associated with a cell of a multiresolution grid and the vehicle may determine multiresolution planes associated with the multiresolution space. For example, the vehicle may determine a first plane associated with a first resolution level, in addition to determining a plurality of planes associated with a second resolution level. The vehicle may determine the validity of the plane(s) at the various resolution levels. Based on the validity of the planes, the vehicle may connect the valid planes that are associated with the ground surface.

Voxel based multiresolution ground plane detection

Techniques for collision avoidance using an object contour are discussed. A trajectory associated with a vehicle may be received. Sensor data can be received from a sensor associated with the vehicle. A bounding contour may be determined and associated with an object represented in the sensor data. Based on the trajectory, a simulated position of the vehicle can be determined. Additionally, a predicted position of the bounding contour can be determined. Based on the simulated position of the vehicle and the predicted position of the bounding contour, a distance between the vehicle and the object may be determined. An action can be performed based on the distance between the vehicle and the object.

Collision avoidance using an object contour

Techniques for determining an object contour are discussed. Depth data associated with an object may be received. The depth data, such as lidar data, can be projected onto a two-dimensional plane. A first convex hull may be determined based on the projected lidar data. The first convex hull may include a plurality of boundary edges. A longest boundary edge, having a first endpoint and a second endpoint, can be determined. An angle can be determined based on the first endpoint, the second endpoint, and an interior point in the interior of the first convex hull. The longest boundary edge may be replaced with a first segment based on the first endpoint and the interior point, and a second segment based on the interior point and the second endpoint. An updated convex hull can be determined based on the first segment and the second segment.

Object contour determination

Techniques for identifying curbs are discussed herein. For instance, a vehicle may generate sensor data using one or more sensors, where the sensor data represents points associated with a driving surface and a sidewalk. The vehicle may then quantize the points into distance bins that are located laterally along the driving direction of the vehicle in order to generate spatial lines. Next, the vehicle may determine separation points for the spatial lines, where the separation points are configured to separate the points associated with the driving surface from the points associated with the sidewalk. The vehicle may then generate, using the separation points, a curve that represents the curb between the driving surface and the sidewalk. This way, the vehicle may use the curve while navigating, such as to avoid the curb and/or stop at a location that is proximate to the curb.

Techniques for identifying curbs

Systems, methods, and apparatuses described herein are directed to performing segmentation on voxels representing three-dimensional data to identify static and dynamic objects. LIDAR data may be captured by a perception system for an autonomous vehicle and represented in a voxel space. Operations may include determining a drivable surface by parsing individual voxels to determine an orientation of a surface normal of a planar approximation of the voxelized data relative to a reference direction. Clustering techniques can be used to grow a ground plane including a plurality of locally flat voxels. Ground plane data can be set aside from the voxel space, and the remaining voxels can be clustered to determine objects. Voxel data can be analyzed over time to determine dynamic objects. Segmentation information associated with ground voxels, static object, and dynamic objects can be provided to a tracker and/or planner in conjunction with operating the autonomous vehicle.

Voxel based ground plane estimation and object segmentation

Techniques for detecting an object in an environment and determining a probability that the object is a cloud of particulate matter. The cloud of particulate matter may include steam (e.g., emitted from a man-hole cover, a dryer exhaust port, etc.), exhaust from a vehicle (e.g., car, truck, motorcycle, etc.), environmental gases (e.g., resulting from sublimation, fog, evaporation, etc.), a cloud of dust, water splashing, blowing leaves, or other types of particulate matter that may be located in the environment of the vehicle and may not impact driving behavior (e.g., an autonomous vehicle may safely pass through the particulate matter without impact to the platform). A vehicle computing system may determine the probability that the object is a cloud of particulate matter and may control the vehicle based on the probability.

Identification of particulate matter in sensor data

Techniques for controlling a vehicle based on height data and/or classification data being determined utilizing multi-channel image data are discussed herein. The vehicle can capture lidar data as it traverses an environment. The lidar data can be associated with a voxel space as three-dimensional data. Semantic information can be determined and associated with the lidar data and/or the three-dimensional voxel space. A multi-channel input image can be determined based on the three-dimensional voxel space and input into a machine learned (ML) model. The ML model can output data to determine height data and/or classification data associated with a ground surface of the environment. The height data and/or classification data can be utilized to determine a mesh associated with the ground surface. The mesh can be used to control the vehicle and/or determine additional objects proximate the vehicle.

Estimating ground height based on lidar data

A vehicle can include various sensors to detect objects in an environment. Sensor data can be captured by a perception system in a vehicle and represented in a voxel space. Operations may include analyzing the data from a top-down perspective. From this perspective, techniques can associate and generate masks that represent objects in the voxel space. Through manipulation of the regions of the masks, the sensor data and/or voxels associated with the masks can be clustered or otherwise grouped to segment data associated with the objects.

Growing community of inventors

Menlo Park, CA, United States of America

Zeng Wang