Nvidia Corporation

Kla Tencor Corporation

Argo Ai Gmbh

Google Inc.

Volkswagen Group of America Investments, LLC

Lytro, Inc.

Gang Pan

Yue Wu

Minwoo Park

Xxx Xinjilefu

Kang Wang

Abdurrahman Sezginer

Cheng-Chieh Yang

Bing Li

Jon Karafin

Thomas Nonn

Tao V Fu

Kunal Anil Desai

Sangmin Oh

Joachim Pehserl

Tae Eun Choe

Ankit Kumar Jain

Jie Tan

Dong Zhang

Samuel Rupp Ogden

Baris Evrim Demiroz

Philippe Babin

Weizhao Shao

Brendan Christopher Byrne

Ayon Sen

Feng Zhao

Zejing Wang

Yifang Xu

Julio Cesar Hernandez Zaragoza

Tao Fu

Liwen Lin

Manu Sethi

Anurag Singh

Sakib Ar Rahman

In various examples, detecting occluded objects within images or other sensor data representations for autonomous or semi-autonomous systems and applications is described herein. Systems and methods described herein may determine when objects are occluded at portions of images using various techniques. For example, an image may be processed in order to determine classifications associated with objects depicted by the image and, the classifications, along with labels that are projected on the image using a map, may then be used to determine whether one or more of the objects are occluded in the image. For another example, a map may be used to determine first distances to points within an environment and a point cloud may be used to determine second distances to the points within the environment. The distances may then be used to determine whether one or more objects are occluded within the image.

Detecting occluded objects within images for autonomous systems and applications

In examples, system and methods for stereo disparity based hazard detection for autonomous machine applications are presented. Example embodiments may assist an ego-machine in detecting hazards within its path of travel. The systems and methods may use disparity between a stereo pair of images to generate a baseline path disparity model and further identify hazards from detected disparities that deviate from that path disparity model. A disparity map for the image pair is constructed in which each pixel represents a disparity for a corresponding element of the image captured. Blockwise division may be optionally used to subdivide the disparity map into a plurality of smaller disparity maps, each corresponding to a block of pixels of the disparity map. A V-space disparity map, where a first axis corresponds to disparity values and the second axis corresponds to pixel rows, may be used to simplify estimation of the path disparity model.

Detecting hazards based on disparity maps using computer vision for autonomous machine systems and applications

In various examples, sensor configuration for autonomous or semi-autonomous systems and applications is described. Systems and methods are disclosed that may use image feature correspondences between camera images along with an assumption that image features are locally planar to determine parameters for calibrating an image sensor with a LiDAR sensor and/or another image sensor. In some examples, an optimization problem is constructed that attempts to minimize a geometric loss function, where the geometric loss function encodes the notion that corresponding image features are views of a same point on a locally planar surface (e.g., a surfel or mesh) that is constructed from LiDAR data generated using a LiDAR sensor. In some examples, performing such processes to determine the calibration parameters may remove structure estimation from the optimization problem.

Sensor calibration for autonomous systems and applications

In various examples, a hazard detection system plots hazard indicators from multiple detection sensors to grid cells of an occupancy grid corresponding to a driving environment. For example, as the ego-machine travels along a roadway, one or more sensors of the ego-machine may capture sensor data representing the driving environment. A system of the ego-machine may then analyze the sensor data to determine the existence and/or location of the one or more hazards within an occupancy grid—and thus within the environment. When a hazard is detected using a respective sensor, the system may plot an indicator of the hazard to one or more grid cells that correspond to the detected location of the hazard. Based, at least in part, on a fused or combined confidence of the hazard indicators for each grid cell, the system may predict whether the corresponding grid cell is occupied by a hazard.

Hazard detection using occupancy grids for autonomous systems and applications

A system and method for performing visual odometry is disclosed. In aspects, the system implements methods to generate an image pyramid based on an input image received. A refined pose prior information representing a location and orientation of the autonomous vehicle can be generated based on one or more images of the image pyramid. One or more seed points can be selected from the one or more images of the image pyramid. One or more refined seed representing the one or more seed points with added depth values can be generated. One or more scene points can be generated based on the one or more refined seed points. A point cloud can be generated based on the one or more scene points.

Heterogeneous multi-threaded visual odometry in autonomous driving

In various examples, a hazard detection system fuses outputs from multiple sensors over time to determine a probability that a stationary object or hazard exists at a location. The system may then use sensor data to calculate a detection bounding shape for detected objects and, using the bounding shape, may generate a set of particles, each including a confidence value that an object exists at a corresponding location. The system may then capture additional sensor data by one or more sensors of the ego-machine that are different from those used to capture the first sensor data. To improve the accuracy of the confidences of the particles, the system may determine a correspondence between the first sensor data and the additional sensor data (e.g., depth sensor data), which may be used to filter out a portion of the particles and improve the depth predictions corresponding to the object.

Particle-based hazard detection for autonomous machine

In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated using a simulated environment. For example, a simulation may be run to simulate a virtual world or environment, render frames of virtual sensor data (e.g., images), and generate corresponding depth maps and segmentation masks (identifying a component of the simulated environment such as a road). To generate input training data, 3D structure estimation may be performed on a rendered frame to generate a representation of a 3D surface structure of the road. To generate corresponding ground truth training data, a corresponding depth map and segmentation mask may be used to generate a dense representation of the 3D surface structure.

3D surface structure estimation using neural networks for autonomous systems and applications

Disclosed herein are system, method, and computer program product embodiments for automated autonomous vehicle pose validation. An embodiment operates by generating a range image from a point cloud solution comprising a pose estimate for an autonomous vehicle. The embodiment queries the range image for predicted ranges and predicted class labels corresponding to lidar beams projected into the range image. The embodiment generates a vector of features from the range image. The embodiment compares a plurality of values to the vector of features using a binary classifier. The embodiment validates the autonomous vehicle pose based on the comparison of the plurality of values to the vector of features using the binary classifier.

Automated vehicle pose validation

In various examples, a 3D surface structure such as the 3D surface structure of a road (3D road surface) may be observed and estimated to generate a 3D point cloud or other representation of the 3D surface structure. Since the estimated representation may be sparse, a deep neural network (DNN) may be used to predict values for a dense representation of the 3D surface structure from the sparse representation. For example, a sparse 3D point cloud may be projected to form a sparse projection image (e.g., a sparse 2D height map), which may be fed into the DNN to predict a dense projection image (e.g., a dense 2D height map). The predicted dense representation of the 3D surface structure may be provided to an autonomous vehicle drive stack to enable safe and comfortable planning and control of the autonomous vehicle.

3D surface reconstruction with point cloud densification using deep neural networks for autonomous systems and applications

In various examples, a 3D surface structure such as the 3D surface structure of a road (3D road surface) may be observed and estimated to generate a 3D point cloud or other representation of the 3D surface structure. Since the representation may be sparse, one or more densification techniques may be applied to densify the representation of the 3D surface structure. For example, the relationship between sparse and dense projection images (e.g., 2D height maps) may be modeled with a Markov random field, and Maximum a Posterior (MAP) inference may be performed using a corresponding joint probability distribution to estimate the most likely dense values given the sparse values. The resulting dense representation of the 3D surface structure may be provided to an autonomous vehicle drive stack to enable safe and comfortable planning and control of the autonomous vehicle.

3D surface reconstruction with point cloud densification using artificial intelligence for autonomous systems and applications

In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated from real-world data. For example, one or more vehicles may collect image data and LiDAR data while navigating through a real-world environment. To generate input training data, 3D surface structure estimation may be performed on captured image data to generate a sparse representation of a 3D surface structure of interest (e.g., a 3D road surface). To generate corresponding ground truth training data, captured LiDAR data may be smoothed, subject to outlier removal, subject to triangulation to filling missing values, accumulated from multiple LiDAR sensors, aligned with corresponding frames of image data, and/or annotated to identify 3D points on the 3D surface of interest, and the identified 3D points may be projected to generate a dense representation of the 3D surface structure.

Using neural networks for 3D surface structure estimation based on real-world data for autonomous systems and applications

A system and method for performing visual localization is disclosed. In aspects, the system implements methods to generate a global point cloud, the global point cloud representing a plurality of point clouds. The global point cloud can be mapped to a prior map information to locate a position of an autonomous vehicle, the prior map information representing pre-built geographic maps. The position of the autonomous vehicle can be estimated based on applying sensor information obtained from sensors and software of the autonomous vehicle to the mapped global point cloud.

Visual localization against a prior map

In various examples, to support training a deep neural network (DNN) to predict a dense representation of a 3D surface structure of interest, a training dataset is generated using a parametric mathematical modeling. A variety of synthetic 3D road surfaces may be generated by modeling a 3D road surface using varied parameters to simulate changes in road direction and lateral surface slope. In an example embodiment, a synthetic 3D road surface may be created by modeling a longitudinal 3D curve and expanding the longitudinal 3D curve to a 3D surface, and the resulting synthetic 3D surface may be sampled to form a synthetic ground truth projection image (e.g., a 2D height map). To generate corresponding input training data, a known pattern that represents which pixels may remain unobserved during 3D structure estimation may be generated and applied to a ground truth projection image to simulate a corresponding sparse projection image.

Disclosed herein are system and method embodiments to implement a validation of an SfM map. An embodiment operates by receiving a motion-generated map corresponding to a digital image, generating a first depth map, wherein the first depth map comprises depth information for one or more triangulated points located within the motion generated image. The embodiment further receives a light detection and ranging (lidar) generated point cloud including at least a portion of the one or more triangulated points, splats the lidar point cloud proximate to the portion of the one or more triangulated points and generates a second depth map for the portion and identifies an incorrect triangulated point, of the one or more triangulated points, based on comparing the first depth information to the second depth information. The incorrect triangulated points may be removed from the SfM map or marked with a low degree of confidence.

Growing community of inventors

Fremont, CA, United States of America

Gang Pan