Google Inc.

Waymo LLC

Zoox, Inc.

Like.com

Leland Stanford Junior University

Dragomir Dimitrov Anguelov

Salih Burak Gokturk

Vincent Olivier Vanhoucke

Munjal Shah

Diem Thanh Vu

Kuang-chih Lee

Azhar H Khan

Danny Yang

Lorenzo Torresani

Subhasis Das

Zeng Wang

Yin Zhou

Xiaofan Lin

Navneet Dalal

Pei Sun

Baris Sumengen

Yuning Chai

Benjamin Sapp

John Patrick Flynn

Jiyang Gao

Ehud Benyamin Rivlin

Congcong Li

Bertrand Robert Douillard

William Anthony Silva

Mayank Bansal

Ashwin Devendrappa Tengli

Benjamin Isaac Zwiebel

Juhana Kangaspunta

Jesse Sol Levinson

Vijay Vasudevan

David Pfeiffer

Scott E Lininger

Junhua Mao

Jiquan Ngiam

Dumitru Erhan

Ehud Rivlin

Allan Zelener

Hang Zhao

Ruizhongtai Qi

Arsalan Mousavian

Rafael Spring

Alexander Gorban

Weiyue Wang

Hartmut Neven

Yang Song

Hartwig Adam

Jay Yagnik

Abhijit S Ogale

Yu Ouyang

Yu Zhang

Daniel Joseph Filip

Colin Andrew Braley

Xiao Meng Zhang

Christian Szegedy

Chen Sun

Mingxing Tan

Reza Mahjourian

Christian Frueh

Anand Pillai

Cordelia Luise Schmid

Ashesh Jain

James William Philbin

Sabeek Mani Pradhan

Sameer Agarwal

Tencia Lee

Henrik Kretzschmar

Zijian Guo

Xinwei Shi

Christoph Sprunk

Yi Shen

Mark Cummins

Qi-xing Huang

Andre Liang Cornman

Danfei Xu

Yue Shen

Ting Liu

Jingxiao Zheng

Zhangjie Cao

David Petrou

Sebastian Thrun

Andrew Rabinovich

Brian McClendon

Alexander Patrick McCauley

Khaled Refaat

Alexander Toshev

David R Martin

Nitesh Shroff

Balakrishnan Varadarajan

Brice Rebsamen

Lo Po Tsui

Sean Rafferty

Xin Huang

Daphne Koller

Benjamin James Caine

Yulai Shen

Chenxi Liu

Jinkyu Kim

Jeffrey Carlson

Jeffrey Sham

Xiangxin Zhu

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using context-sensitive fusion.

Agent trajectory prediction using context-sensitive fusion

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for estimating a 3-D pose of an object of interest from image and point cloud data. In one aspect, a method includes obtaining an image of an environment; obtaining a point cloud of a three-dimensional region of the environment; generating a fused representation of the image and the point cloud; and processing the fused representation using a pose estimation neural network and in accordance with current values of a plurality of pose estimation network parameters to generate a pose estimation network output that specifies, for each of multiple keypoints, a respective estimated position in the three-dimensional region of the environment.

Multi-modal 3-D pose estimation

A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle up to a current time point. The system identifies a plurality of initial target locations, and generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent. For each target location in a first subset of the target locations, the system generates a predicted future trajectory for the agent given that the target location is the intended final location for the future trajectory. The system further selects, as likely future trajectories of the agent, one or more of the predicted future trajectories.

Agent trajectory prediction using target locations

Methods, systems, and apparatus for processing point clouds using neural networks to perform a machine learning task. In one aspect, a system comprises one or more computers configured to obtain a set of point clouds captured by one or more sensors. Each point cloud includes a respective plurality of three-dimensional points. The one or more computers assign the three-dimensional points to respective voxels in a voxel grid, where the grid of voxels includes non-empty voxels to which one or more points are assigned and empty voxels to which no points are assigned. For each non-empty voxel, the one or more computers generate initial features based on the points that are assigned to the non-empty voxel. The one or more computers generate multi-scale features of the voxel grid, and the one or more computers generate an output for a point cloud processing task using the multi-scale features of the voxel grid.

Performing point cloud tasks using multi-scale features generated through self-attention

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting the future movement of agents in an environment. In particular, the future movement is predicted through occupancy flow fields that specify, for each future time point in a sequence of future time points and for each agent type in a set of one or more agent types: an occupancy prediction for the future time step that specifies, for each grid cell, an occupancy likelihood that any agent of the agent type will occupy the grid cell at the future time point, and a motion flow prediction that specifies, for each grid cell, a motion vector that represents predicted motion of agents of the agent type within the grid cell at the future time point.

Predicting the future movement of agents in an environment using occupancy flow fields

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for generating and editing object track labels for objects detected in video data. One of the methods includes obtaining a video segment comprising multiple image frames associated with multiple time points; obtaining object track data specifying a set of object tracks; providing, for presentation to a user, a user interface for modifying the object track data, the user interface displaying object timeline representations of the object tracks; receiving one or more user inputs that indicate one or more modifications to the object timeline representations; updating the object timeline representations displayed in the timeline display area; and updating the object track data according to the updated object timeline representations.

Time-line based object tracking annotation

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating a behavior prediction system.

Evaluating multi-modal trajectory predictions for autonomous driving

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generated simulated sensor data. One of the methods includes obtaining a surfel map generated from sensor observations of a real-world environment and generating, for each surfel in the surfel map, a respective grid having a plurality of grid cells, wherein each grid has an orientation matching an orientation of a corresponding surfel, and wherein each grid cell within each grid is assigned a respective color value. For a simulated location within a simulated representation of the real-world environment, a textured surfel rendering is generated, including combining color information from grid cells visible from the simulated location within the simulated representation of the real-world environment.

Generating environmental data

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing object detection. The system obtains a respective range image corresponding to each point cloud in a set of point clouds captured by one or more sensors. The system processes each range image using a segmentation neural network to generate range image features and a segmentation output. The system generates a feature representation of the set of point clouds from only the feature representations of the foreground points. The system processes the feature representation of the set of point clouds using a prediction neural network to generate a prediction characterizing the set of point clouds.

Efficient three-dimensional object detection from point clouds

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Agent trajectory prediction using anchor trajectories

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing three-dimensional auto-labeling on sensor data. The system obtains a sensor data segment that includes a temporal sequence of three-dimensional point clouds generated from sensor readings of an environment by one or more sensors. The system identifies, from the sensor data segment, (i) a plurality of object tracks that each corresponds to a different object in the environment and (ii) for each object track, respective initial three-dimensional regions in each of one or more of the point clouds in which the corresponding object appears. The system generates, for each object track, extracted object track data that includes at least the points in the respective initial three-dimensional regions for the object track. The system further generates, for each object track and from the extracted object track data for the object track, an auto labeling output that defines respective refined three-dimensional regions in each of the one or more point clouds.

Object-centric three-dimensional auto labeling of point cloud data

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining estimated ground truth object keypoint labels for sensor readings of objects. In one aspect, a method comprises obtaining a plurality of sets of label data for a sensor reading of an object; obtaining respective quality control data corresponding to each of the plurality of sets of label data, the respective quality control data comprising: data indicating whether the labeled location of the first object keypoint in the corresponding set of label data is accurate; and determining an estimated ground truth location for the first object keypoint in the sensor data keypoint from (i) the labeled locations that were indicated as accurate by the corresponding quality control data and (ii) not from the labeled locations that were indicated as not accurate by the corresponding quality control data.

Estimating ground truth object keypoint labels for sensor readings

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting occupancies of agents. One of the methods includes obtaining scene data characterizing a current scene in an environment; and processing a neural network input comprising the scene data using a neural network to generate a neural network output, wherein: the neural network output comprises respective occupancy outputs corresponding to a plurality of agent types at one or more future time points; the occupancy output for each agent type at a first future time point comprises respective occupancy probabilities for a plurality of locations in the environment; and in the occupancy output for each agent type at the first future time point, the respective occupancy probability for each location characterizes a likelihood that an agent of the agent type will occupy the location at the first future time point.

Behavior prediction of surrounding agents

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using temporal-spatial interaction predictions.

Multi-modal multi-agent trajectory prediction

Growing community of inventors

San Francisco, CA, United States of America

Dragomir Dimitrov Anguelov