Nvidia Corporation

Microsoft Technology Licensing, LLC

Telefonaktiebolaget Lm Ericsson (publ)

Sarnoff Corporation

L-3 Communications Corporation

David Nister

Minwoo Park

Sangmin Oh

Ibrahim Eden

Neda Cvijetic

Ruchi Bhargava

Vaibhav Thukral

Junghyun Kwon

Berta Rodriguez Hervas

Zhaoting Ye

Nikolai Smolyanskiy

Michael Kroepfl

Hae-Jong Seo

Joachim Pehserl

Julia C Ng

Yizhou Wang

Tilman Wekel

Alexander A Popov

Yilin Yang

Trung Pham

Birgit Henke

Ryan Oldja

Pekka Janis

Bala Siva Sashank Jujjavarapu

Ke Chen

Djordje Nijemcevic

Tommi Koivisto

Michael Grabner

Christopher Maurice Mei

Amir Akbarzadeh

Rotem Aviv

Daniel Herrera Castro

Ishwar Kulkarni

Bernard Kerr

Yue Wu

Cheng-Chieh Yang

Alexey Kamenev

Morgan Kolya Venable

Gary D Hicok

Urs A Muller

Dongwoo Lee

Weerapan Wilairat

Milan Vugdelija

Matthew Sammis Ashman

Ivan Stojiljkovic

Xiaolin Lin

David Wehr

David Douglas Bohn

Scott Charles McEldowney

Nathan Ackerman

Vivek Pradeep

Michael Brian Cox

Stephen Charles Hsu

Gerhard A Schneider

Ratin Kumar

Quentin Simon Charles Miller

Xinye Lou

Steve Robbins

Wolf Kienzle

Charilaos Christopoulos

Gordon Grigor

Mudit Agrawal

Bernhard Firner

Magdalena Vukosavljevic

Justin Ebert

Miguel Sainz

Zongyi Yang

Timo Roman

Piotr Dollar

Davide Onofrio

Eyal Ofek

Keith James Hanna

Wenyi Zhao

Harpreet Singh Sawhney

Rakesh Kumar

Daniel Buchmueller

Timothy Andrew Large

Supun Samarasekera

Derek L Knee

Nigel Stuart Keam

Andrew Clark Goris

Kazuhito Koishida

Sudipta Narayan Sinha

James Russell Bergen

Blaise Hilary Aguera-Arcas

Drew Edward Steedly

Frederik Schaffalitzky

Regan Blythe Towal

Manoj Aggarwal

Karl Friedrich Greb

Aydin Arpa

Scott Vernon Fynn

John Eldridge

Georg Klein

James Peele Terrell, Jr

John George Mathieson

Ram Ganapathi

Jonathan Sweedler

Youding Zhu

Ashok Srinivasan

A deep neural network(s) (DNN) may be used to perform panoptic segmentation by performing pixel-level class and instance segmentation of a scene using a single pass of the DNN. Generally, one or more images and/or other sensor data may be stitched together, stacked, and/or combined, and fed into a DNN that includes a common trunk and several heads that predict different outputs. The DNN may include a class confidence head that predicts a confidence map representing pixels that belong to particular classes, an instance regression head that predicts object instance data for detected objects, an instance clustering head that predicts a confidence map of pixels that belong to particular instances, and/or a depth head that predicts range values. These outputs may be decoded to identify bounding shapes, class labels, instance labels, and/or range values for detected objects, and used to enable safe path planning and control of an autonomous vehicle.

Deep neural network for segmentation of road scenes and animate object instances for autonomous driving applications

In various examples, live perception from sensors of a vehicle may be leveraged to detect and classify intersections in an environment of a vehicle in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute various outputs—such as bounding box coordinates for intersections, intersection coverage maps corresponding to the bounding boxes, intersection attributes, distances to intersections, and/or distance coverage maps associated with the intersections. The outputs may be decoded and/or post-processed to determine final locations of, distances to, and/or attributes of the detected intersections.

Intersection detection and classification in autonomous machine applications

Embodiments of the present disclosure relate to generating RADAR (RAdio Detection And Ranging) point clouds based on RADAR data obtained from one or more RADAR sensors disposed on one or more ego-machines. In these or other embodiments, the RADAR point clouds may be used to generate map data. Additionally or alternatively, the RADAR point clouds may be used for performing localization.

Sensor data point cloud generation for map creation and localization for autonomous systems and applications

A neural network may be used to determine corner points of a skewed polygon (e.g., as displacement values to anchor box corner points) that accurately delineate a region in an image that defines a parking space. Further, the neural network may output confidence values predicting likelihoods that corner points of an anchor box correspond to an entrance to the parking spot. The confidence values may be used to select a subset of the corner points of the anchor box and/or skewed polygon in order to define the entrance to the parking spot. A minimum aggregate distance between corner points of a skewed polygon predicted using the CNN(s) and ground truth corner points of a parking spot may be used simplify a determination as to whether an anchor box should be used as a positive sample for training.

Object detection using skewed polygons suitable for parking space detection

In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space, in both highway and urban scenarios. RADAR detections may be accumulated, ego-motion-compensated, orthographically projected, and fed into a neural network(s). The neural network(s) may include a common trunk with a feature extractor and several heads that predict different outputs such as a class confidence head that predicts a confidence map and an instance regression head that predicts object instance data for detected objects. The outputs may be decoded, filtered, and/or clustered to form bounding shapes identifying the location, size, and/or orientation of detected object instances. The detected object instances may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

Deep neural network for detecting obstacle instances using radar sensors in autonomous machine applications

In various examples, sensor data recorded in the real-world may be leveraged to generate transformed, additional, sensor data to test one or more functions of a vehicle—such as a function of an AEB, CMW, LDW, ALC, or ACC system. Sensor data recorded by the sensors may be augmented, transformed, or otherwise updated to represent sensor data corresponding to state information defined by a simulation test profile for testing the vehicle function(s). Once a set of test data has been generated, the test data may be processed by a system of the vehicle to determine the efficacy of the system with respect to any number of test criteria. As a result, a test set including additional or alternative instances of sensor data may be generated from real-world recorded sensor data to test a vehicle in a variety of test scenarios.

Simulating realistic test data from transformed real-world sensor data for autonomous machine

In various examples, a current claimed set of points representative of a volume in an environment occupied by a vehicle at a time may be determined. A vehicle-occupied trajectory and at least one object-occupied trajectory may be generated at the time. An intersection between the vehicle-occupied trajectory and an object-occupied trajectory may be determined based at least in part on comparing the vehicle-occupied trajectory to the object-occupied trajectory. Based on the intersection, the vehicle may then execute the first safety procedure or an alternative procedure that, when implemented by the vehicle when the object implements the second safety procedure, is determined to have a lesser likelihood of incurring a collision between the vehicle and the object than the first safety procedure.

Safety procedure analysis for obstacle avoidance in autonomous vehicles

In various examples, one or more output channels of a deep neural network (DNN) may be used to determine assignments of obstacles to paths. To increase the accuracy of the DNN, the input to the DNN may include an input image, one or more representations of path locations, and/or one or more representations of obstacle locations. The system may thus repurpose previously computed information—e.g., obstacle locations, path locations, etc.—from other operations of the system, and use them to generate more detailed inputs for the DNN to increase accuracy of the obstacle to path assignments. Once the output channels are computed using the DNN, computed bounding shapes for the objects may be compared to the outputs to determine the path assignments for each object. Additionally, a machine may perform control operations based at least on the path assignments.

Obstacle to path assignment and performance of control operations based on assignment for autonomous systems and applications

In various examples, live perception from sensors of a vehicle may be leveraged to detect and classify intersection contention areas in an environment of a vehicle in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute outputs—such as signed distance functions—that may correspond to locations of boundaries delineating intersection contention areas. The signed distance functions may be decoded and/or post-processed to determine instance segmentation masks representing locations and classifications of intersection areas or regions. The locations of the intersections areas or regions may be generated in image-space and converted to world-space coordinates to aid an autonomous or semi-autonomous vehicle in navigating intersections according to rules of the road, traffic priority considerations, and/or the like.

Intersection region detection and classification for autonomous machine applications

One or more embodiments of the present disclosure relate to generation of map data. In these or other embodiments, the generation of the map data may include determining whether objects indicated by the sensor data are static objects or dynamic objects. Additionally or alternatively, sensor data may be removed or included in the map data based on determinations as to whether it corresponds to static objects or dynamic objects.

Sensor data based map creation for autonomous systems and applications

In various examples, a three-dimensional (3D) intersection structure may be predicted using a deep neural network (DNN) based on processing two-dimensional (2D) input data. To train the DNN to accurately predict 3D intersection structures from 2D inputs, the DNN may be trained using a first loss function that compares 3D outputs of the DNN—after conversion to 2D space—to 2D ground truth data and a second loss function that analyzes the 3D predictions of the DNN in view of one or more geometric constraints—e.g., geometric knowledge of intersections may be used to penalize predictions of the DNN that do not align with known intersection and/or road structure geometries. As such, live perception of an autonomous or semi-autonomous vehicle may be used by the DNN to detect 3D locations of intersection structures from 2D inputs.

Three-dimensional intersection structure prediction for autonomous driving applications

In various examples, an end-to-end perception evaluation system for autonomous and semi-autonomous machine applications may be implemented to evaluate how the accuracy or precision of outputs of machine learning models—such as deep neural networks (DNNs)—impact downstream performance of the machine when relied upon. For example, decisions computed by the system using ground truth output types may be compared to decisions computed by the system using the perception outputs. As a result, discrepancies in downstream decision making of the system between the ground truth information and the perception information may be evaluated to either aid in updating or retraining of the machine learning model or aid in generating more accurate or precise ground truth information.

End-to-end evaluation of perception systems for autonomous systems and applications

In various examples, object fence corresponding to objects detected by an ego-vehicle may be used to determine overlap of the object fences with lanes on a driving surface. A lane mask may be generated corresponding to the lanes on the driving surface, and the object fences may be compared to the lanes of the lane mask to determine the overlap. Where an object fence is located in more than one lane, a boundary scoring approach may be used to determine a ratio of overlap of the boundary fence, and thus the object, with each of the lanes. The overlap with one or more lanes for each object may be used to determine lane assignments for the objects, and the lane assignments may be used by the ego-vehicle to determine a path or trajectory along the driving surface.

Lane mask generation for autonomous machine

In various examples, systems and methods are disclosed that preserve rich spatial information from an input resolution of a machine learning model to regress on lines in an input image. The machine learning model may be trained to predict, in deployment, distances for each pixel of the input image at an input resolution to a line pixel determined to correspond to a line in the input image. The machine learning model may further be trained to predict angles and label classes of the line. An embedding algorithm may be used to train the machine learning model to predict clusters of line pixels that each correspond to a respective line in the input image. In deployment, the predictions of the machine learning model may be used as an aid for understanding the surrounding environment—e.g., for updating a world model—in a variety of autonomous machine applications.

Regression-based line detection for autonomous driving machines

In various examples, a safety decomposition architecture for autonomous machine applications is presented that uses two or more individual safety assessments to satisfy a higher safety integrity level (e.g., ASIL D). For example, a behavior planner may be used as a primary planning component, and a collision avoidance feature may be used as a diverse safety monitoring component—such that both may redundantly and independently prevent violation of safety goals. In addition, robustness of the system may be improved as single point and systematic failures may be avoided due to the requirement that two independent failures—e.g., of the behavior planner component and the collision avoidance component—occur simultaneously to cause a violation of the safety goals.

Safety decomposition architecture for autonomous machine applications

Growing community of inventors

Bellevue, WA, United States of America

David Nister