Waymo LLC

Gm Cruise Holdings LLC

Yuning Chai

Dragomir Dimitrov Anguelov

Benjamin Sapp

Pei Sun

Xiao Meng Zhang

Congcong Li

Vijay Vasudevan

Mayank Bansal

Chen Sun

Jiquan Ngiam

Yin Zhou

Jiyang Gao

Hang Zhao

Cordelia Luise Schmid

Tian Lan

Benjamin James Caine

Yi Shen

Yue Shen

Weiyue Wang

Zhifeng Chen

Jonathon Shlens

Patrick An Phu Nguyen

Mingxing Tan

Ouais Alsharif

Dumitru Erhan

Reza Mahjourian

Wei Han

Zhao Chen

Henrik Kretzschmar

Sean Rafferty

Brandon Chauloon Yang

Jinkyu Kim

Zhenpei Yang

Hongge Chen

Xi Yi

The present disclosure provides solutions for generating synthetic three-dimensional (3D) objects for testing autonomous vehicle (AV) object detectors, such as AV Light Detection and Ranging (LiDAR) detectors. A process of the disclosed technology can include steps for receiving a first set of object shape data corresponding to a baseline object, permuting the first set of object shape data to generate a second set of object shape data, and providing the second set of object shape data to a differentiable pipeline to generate an output scene, wherein the output scene contains an optimized object that is based on the second set of object shape data. The process can further include steps for determining, using a detector, if the optimized object can be identified in the output scene. Systems and machine-readable media are also provided.

Pipeline for generating synthetic point cloud data

A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle up to a current time point. The system identifies a plurality of initial target locations, and generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent. For each target location in a first subset of the target locations, the system generates a predicted future trajectory for the agent given that the target location is the intended final location for the future trajectory. The system further selects, as likely future trajectories of the agent, one or more of the predicted future trajectories.

Agent trajectory prediction using target locations

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting the future movement of agents in an environment. In particular, the future movement is predicted through occupancy flow fields that specify, for each future time point in a sequence of future time points and for each agent type in a set of one or more agent types: an occupancy prediction for the future time step that specifies, for each grid cell, an occupancy likelihood that any agent of the agent type will occupy the grid cell at the future time point, and a motion flow prediction that specifies, for each grid cell, a motion vector that represents predicted motion of agents of the agent type within the grid cell at the future time point.

Predicting the future movement of agents in an environment using occupancy flow fields

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generated simulated sensor data. One of the methods includes obtaining a surfel map generated from sensor observations of a real-world environment and generating, for each surfel in the surfel map, a respective grid having a plurality of grid cells, wherein each grid has an orientation matching an orientation of a corresponding surfel, and wherein each grid cell within each grid is assigned a respective color value. For a simulated location within a simulated representation of the real-world environment, a textured surfel rendering is generated, including combining color information from grid cells visible from the simulated location within the simulated representation of the real-world environment.

Generating environmental data

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing object detection. The system obtains a respective range image corresponding to each point cloud in a set of point clouds captured by one or more sensors. The system processes each range image using a segmentation neural network to generate range image features and a segmentation output. The system generates a feature representation of the set of point clouds from only the feature representations of the foreground points. The system processes the feature representation of the set of point clouds using a prediction neural network to generate a prediction characterizing the set of point clouds.

Efficient three-dimensional object detection from point clouds

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Agent trajectory prediction using anchor trajectories

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for processing a perspective view range image generated from sensor measurements of an environment. The perspective view range image includes a plurality of pixels arranged in a two-dimensional grid and including, for each pixel, (i) features of one or more sensor measurements at a location in the environment corresponding to the pixel and (ii) geometry information comprising range features characterizing a range of the location in the environment corresponding to the pixel relative to the one or more sensors. The system processes the perspective view range image using a first neural network to generate an output feature representation. The first neural network comprises a first perspective point-set aggregation layer comprising a geometry-dependent kernel.

Processing perspective view range images using neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data representing a sensor measurement of a scene captured by one or more sensors to generate an object detection output that identifies locations of one or more objects in the scene. When deployed within an on-board system of a vehicle, the object detection output that is generated can be used to make autonomous driving decisions for the vehicle with enhanced accuracy.

Object detection in point clouds

Growing community of inventors

San Mateo, CA, United States of America

Yuning Chai