Waymo LLC

Apple Inc.

Samsung Electronics Co., Ltd.

Yin Zhou

Dragomir Dimitrov Anguelov

Pei Sun

Vijay Vasudevan

Jiquan Ngiam

Cuneyt Oncel Tuzel

Ruizhongtai Qi

Alexander Gorban

Yang Song

Yu Ouyang

Yu Zhang

Congcong Li

Junhua Mao

Jiyang Gao

Yuning Chai

Zijian Guo

Xinwei Shi

Andre Liang Cornman

Jingxiao Zheng

Ting Liu

Zhangjie Cao

Russell Y Webb

Zhifeng Chen

Bo Li

Jonathon Shlens

Jerremy Holland

Patrick An Phu Nguyen

Ouais Alsharif

Dumitru Erhan

Wei Han

Jonathan James Currey

Henrik Kretzschmar

Xiaolong Wang

Sean Rafferty

Benjamin James Caine

Brandon Chauloon Yang

Alessandro Giulianelli

Vishwanath A Sindagi

Zhenpei Yang

Luca Ballan

Xi Yi

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for estimating a 3-D pose of an object of interest from image and point cloud data. In one aspect, a method includes obtaining an image of an environment; obtaining a point cloud of a three-dimensional region of the environment; generating a fused representation of the image and the point cloud; and processing the fused representation using a pose estimation neural network and in accordance with current values of a plurality of pose estimation network parameters to generate a pose estimation network output that specifies, for each of multiple keypoints, a respective estimated position in the three-dimensional region of the environment.

Multi-modal 3-D pose estimation

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generated simulated sensor data. One of the methods includes obtaining a surfel map generated from sensor observations of a real-world environment and generating, for each surfel in the surfel map, a respective grid having a plurality of grid cells, wherein each grid has an orientation matching an orientation of a corresponding surfel, and wherein each grid cell within each grid is assigned a respective color value. For a simulated location within a simulated representation of the real-world environment, a textured surfel rendering is generated, including combining color information from grid cells visible from the simulated location within the simulated representation of the real-world environment.

Generating environmental data

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for performing three-dimensional auto-labeling on sensor data. The system obtains a sensor data segment that includes a temporal sequence of three-dimensional point clouds generated from sensor readings of an environment by one or more sensors. The system identifies, from the sensor data segment, (i) a plurality of object tracks that each corresponds to a different object in the environment and (ii) for each object track, respective initial three-dimensional regions in each of one or more of the point clouds in which the corresponding object appears. The system generates, for each object track, extracted object track data that includes at least the points in the respective initial three-dimensional regions for the object track. The system further generates, for each object track and from the extracted object track data for the object track, an auto labeling output that defines respective refined three-dimensional regions in each of the one or more point clouds.

Object-centric three-dimensional auto labeling of point cloud data

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining estimated ground truth object keypoint labels for sensor readings of objects. In one aspect, a method comprises obtaining a plurality of sets of label data for a sensor reading of an object; obtaining respective quality control data corresponding to each of the plurality of sets of label data, the respective quality control data comprising: data indicating whether the labeled location of the first object keypoint in the corresponding set of label data is accurate; and determining an estimated ground truth location for the first object keypoint in the sensor data keypoint from (i) the labeled locations that were indicated as accurate by the corresponding quality control data and (ii) not from the labeled locations that were indicated as not accurate by the corresponding quality control data.

Estimating ground truth object keypoint labels for sensor readings

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating realistic full-scene point clouds. One of the methods includes obtaining an initial scene point cloud characterizing an initial scene in an environment; obtaining, for each of one or more objects, an object point cloud that characterizes the object; and processing a first input comprising the initial scene point cloud and the one or more object point clouds using a first neural network that is configured to process the first input to generate a final scene point cloud that characterizes a transformed scene that has the one or more objects added to the initial scene.

Generating realistic point clouds

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data using dynamic voxelization. When deployed within an on-board system of a vehicle, processing the point cloud data using dynamic voxelization can be used to make autonomous driving decisions for the vehicle with enhanced accuracy, for example by combining representations of point cloud data characterizing a scene from multiple views of the scene.

Processing point clouds using dynamic voxelization

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing point cloud data representing a sensor measurement of a scene captured by one or more sensors to generate an object detection output that identifies locations of one or more objects in the scene. When deployed within an on-board system of a vehicle, the object detection output that is generated can be used to make autonomous driving decisions for the vehicle with enhanced accuracy.

Object detection in point clouds

A method includes obtaining surface samples that represent three-dimensional locations of surfaces of an environment; generating a voxelized representation of the surfaces of the environment in three-dimensional space using the surface samples; obtaining an image that shows the surfaces of the environment; associating each of the surface samples with image information that corresponds to a portion of the image that is spatially correlated with a respective one of the surface samples; determining voxel features for voxels from the voxelized representation based on the surface samples and the image information using a first trained machine learning model, wherein the voxel features each describe three-dimensional shapes present within a respective one of the voxels; and detecting objects based on the voxel features.

Multimodal three-dimensional object detection

A voxel feature learning network receives a raw point cloud and converts the point cloud into a sparse 4D tensor comprising three-dimensional coordinates (e.g. X, Y, and Z) for each voxel of a plurality of voxels and a fourth voxel feature dimension for each non-empty voxel. In some embodiments, convolutional mid layers further transform the 4D tensor into a high-dimensional volumetric representation of the point cloud. In some embodiments, a region proposal network identifies 3D bounding boxes of objects in the point cloud based on the high-dimensional volumetric representation. In some embodiments, the feature learning network and the region proposal network are trained end-to-end using training data comprising known ground truth bounding boxes, without requiring human intervention.

Voxel-based feature learning network

A method for detecting objects in an environment includes obtaining, from one or more sensors, distance measurements from the one or more sensors to portions of the environment, generating a representation of a three-dimensional space using the distance measurements, identifying object features in the representation of the three-dimensional space using a neural network, comparing the object features to pre-defined three-dimensional templates to generate scores that represent correspondence of the object features to the pre-defined three-dimensional templates, and determining a location and a rotational orientation for a three-dimensional object based on the scores.

Three-dimensional object detection

One embodiment provides a method comprising receiving a first input image and a second input image. Each input image comprises a facial image of an individual. For each input image, a first set of facial regions of the facial image is distinguished from a second set of facial regions of the facial image based on a learning based model. The first set of facial regions comprises age-invariant facial features, and the second set of facial regions comprises age-sensitive facial features. The method further comprises determining whether the first input image and the second input images comprise facial images of the same individual by performing face verification based on the first set of facial regions of each input image.

Learning model for salient facial region detection

Growing community of inventors

San Jose, CA, United States of America

Yin Zhou