Nvidia Corporation

Zongyi Yang

Bernhard Firner

Urs A Muller

Mariusz Bojarski

David Nister

Rotem Aviv

Joyjit Daw

Roberto Giuseppe Luca Valenti

Jesse Hong

Haiguang Wen

In various examples, sensor data recorded in the real-world may be leveraged to generate transformed, additional, sensor data to test one or more functions of a vehicle—such as a function of an AEB, CMW, LDW, ALC, or ACC system. Sensor data recorded by the sensors may be augmented, transformed, or otherwise updated to represent sensor data corresponding to state information defined by a simulation test profile for testing the vehicle function(s). Once a set of test data has been generated, the test data may be processed by a system of the vehicle to determine the efficacy of the system with respect to any number of test criteria. As a result, a test set including additional or alternative instances of sensor data may be generated from real-world recorded sensor data to test a vehicle in a variety of test scenarios.

Simulating realistic test data from transformed real-world sensor data for autonomous machine

In various examples, sensor data used to train an MLM and/or used by the MLM during deployment, may be captured by sensors having different perspectives (e.g., fields of view). The sensor data may be transformed—to generate transformed sensor data—such as by altering or removing lens distortions, shifting, and/or rotating images corresponding to the sensor data to a field of view of a different physical or virtual sensor. As such, the MLM may be trained and/or deployed using sensor data captured from a same or similar field of view. As a result, the MLM may be trained and/or deployed—across any number of different vehicles with cameras and/or other sensors having different perspectives—using sensor data that is of the same perspective as the reference or ideal sensor.

Simulating viewpoint transformations for sensor independent scene understanding in autonomous systems

In various examples, image space coordinates of an image from a video may be labeled, projected to determine 3D vehicle space coordinates, then transformed to 3D world space coordinates using known 3D world space coordinates and relative positioning between the coordinate spaces. For example, 3D vehicle space coordinates may be temporally correlated with known 3D world space coordinates measured while capturing the video. The known 3D world space coordinates and known relative positioning between the coordinate spaces may be used to offset or otherwise define a transform for the 3D vehicle space coordinates to world space. Resultant 3D world space coordinates may be used for one or more labeled frames to generate ground truth data. For example, 3D world space coordinates for left and right lane lines from multiple frames may be used to define lane lines for any given frame.

Image to world space transformation for ground-truth generation in autonomous systems and applications

In examples, image data representative of an image of a field of view of at least one sensor may be received. Source areas may be defined that correspond to a region of the image. Areas and/or dimensions of at least some of the source areas may decrease along at least one direction relative to a perspective of the at least one sensor. A downsampled version of the region (e.g., a downsampled image or feature map of a neural network) may be generated from the source areas based at least in part on mapping the source areas to cells of the downsampled version of the region. Resolutions of the region that are captured by the cells may correspond to the areas of the source areas, such that certain portions of the region (e.g., portions at a far distance from the sensor) retain higher resolution than others.

Multi-resolution image patches for predicting autonomous navigation paths

In various examples, sets of testing data may be selected and applied to an MLM such that differences in performance of the MLM in the testing between the sets indicates and may be used to determine whether and/or an extent by which the MLM is trained to rely on artifacts. Training data for the MLM may be generated using a first value of a parameter that defines a value of a characteristic of the training data. For testing, first testing data may be selected that corresponds to a second value of the parameter that shifts the value in a first direction and second testing data may be selected that corresponds to a third value of the parameter that shifts the value in a second direction (e.g., opposite the first direction). Various possible actions may be taken based on results of analyzing the differences in performance.

Measuring the effects of augmentation artifacts on a machine learning network

In various examples, sensor data recorded in the real-world may be leveraged to generate transformed, additional, sensor data to test one or more functions of a vehicle—such as a function of an AEB, CMW, LDW, ALC, or ACC system. Sensor data recorded by the sensors may be augmented, transformed, or otherwise updated to represent sensor data corresponding to state information defined by a simulation test profile for testing the vehicle function(s). Once a set of test data has been generated, the test data may be processed by a system of the vehicle to determine the efficacy of the system with respect to any number of test criteria. As a result, a test set including additional or alternative instances of sensor data may be generated from real-world recorded sensor data to test a vehicle in a variety of test scenarios—including those that may be too dangerous to test in the real-world.

Growing community of inventors

Eatontown, NJ, United States of America

Zongyi Yang