Nvidia Corporation

Microsoft Technology Licensing, LLC

Telefonaktiebolaget Lm Ericsson (publ)

Sarnoff Corporation

L-3 Communications Corporation

David Nister

Minwoo Park

Sangmin Oh

Ibrahim Eden

Neda Cvijetic

Ruchi Bhargava

Vaibhav Thukral

Zhaoting Ye

Junghyun Kwon

Berta Rodriguez Hervas

Nikolai Smolyanskiy

Hae-Jong Seo

Yizhou Wang

Michael Kroepfl

Joachim Pehserl

Yilin Yang

Julia C Ng

Tilman Wekel

Pekka Janis

Ryan Oldja

Birgit Henke

Trung Pham

Bala Siva Sashank Jujjavarapu

Alexander A Popov

Djordje Nijemcevic

Tommi Koivisto

Michael Grabner

Christopher Maurice Mei

Rotem Aviv

Daniel Herrera Castro

Ishwar Kulkarni

Bernard Kerr

Ke Chen

Yue Wu

Cheng-Chieh Yang

Morgan Kolya Venable

Amir Akbarzadeh

Gary D Hicok

Urs A Muller

Dongwoo Lee

Weerapan Wilairat

Milan Vugdelija

Matthew Sammis Ashman

Alexey Kamenev

Ivan Stojiljkovic

Xiaolin Lin

David Wehr

David Douglas Bohn

Scott Charles McEldowney

Nathan Ackerman

Vivek Pradeep

Michael Brian Cox

Gerhard A Schneider

Stephen Charles Hsu

Quentin Simon Charles Miller

Ratin Kumar

Xinye Lou

Steve Robbins

Wolf Kienzle

Charilaos Christopoulos

Gordon Grigor

Mudit Agrawal

Bernhard Firner

Magdalena Vukosavljevic

Justin Ebert

Miguel Sainz

Zongyi Yang

Timo Roman

Piotr Dollar

Jeremy Furtek

Davide Onofrio

Eyal Ofek

Keith James Hanna

Wenyi Zhao

Harpreet Singh Sawhney

Rakesh Kumar

Daniel Buchmueller

Timothy Andrew Large

Supun Samarasekera

Derek L Knee

Nigel Stuart Keam

Andrew Clark Goris

Kazuhito Koishida

Sudipta Narayan Sinha

James Russell Bergen

Blaise Hilary Aguera-Arcas

Drew Edward Steedly

Frederik Schaffalitzky

Regan Blythe Towal

Manoj Aggarwal

Karl Friedrich Greb

Aydin Arpa

Scott Vernon Fynn

John Eldridge

Georg Klein

James Peele Terrell, Jr

John George Mathieson

Ram Ganapathi

Jonathan Sweedler

Ashok Srinivasan

A neural network may be used to determine corner points of a skewed polygon (e.g., as displacement values to anchor box corner points) that accurately delineate a region in an image that defines a parking space. Further, the neural network may output confidence values predicting likelihoods that corner points of an anchor box correspond to an entrance to the parking spot. The confidence values may be used to select a subset of the corner points of the anchor box and/or skewed polygon in order to define the entrance to the parking spot. A minimum aggregate distance between corner points of a skewed polygon predicted using the CNN(s) and ground truth corner points of a parking spot may be used simplify a determination as to whether an anchor box should be used as a positive sample for training.

Object detection using skewed polygons suitable for parking space detection

In various examples, a deep neural network(s) (e.g., a convolutional neural network) may be trained to detect moving and stationary obstacles from RADAR data of a three dimensional (3D) space, in both highway and urban scenarios. RADAR detections may be accumulated, ego-motion-compensated, orthographically projected, and fed into a neural network(s). The neural network(s) may include a common trunk with a feature extractor and several heads that predict different outputs such as a class confidence head that predicts a confidence map and an instance regression head that predicts object instance data for detected objects. The outputs may be decoded, filtered, and/or clustered to form bounding shapes identifying the location, size, and/or orientation of detected object instances. The detected object instances may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

Deep neural network for detecting obstacle instances using radar sensors in autonomous machine applications

In various examples, sensor data recorded in the real-world may be leveraged to generate transformed, additional, sensor data to test one or more functions of a vehicle—such as a function of an AEB, CMW, LDW, ALC, or ACC system. Sensor data recorded by the sensors may be augmented, transformed, or otherwise updated to represent sensor data corresponding to state information defined by a simulation test profile for testing the vehicle function(s). Once a set of test data has been generated, the test data may be processed by a system of the vehicle to determine the efficacy of the system with respect to any number of test criteria. As a result, a test set including additional or alternative instances of sensor data may be generated from real-world recorded sensor data to test a vehicle in a variety of test scenarios.

Simulating realistic test data from transformed real-world sensor data for autonomous machine

In various examples, a current claimed set of points representative of a volume in an environment occupied by a vehicle at a time may be determined. A vehicle-occupied trajectory and at least one object-occupied trajectory may be generated at the time. An intersection between the vehicle-occupied trajectory and an object-occupied trajectory may be determined based at least in part on comparing the vehicle-occupied trajectory to the object-occupied trajectory. Based on the intersection, the vehicle may then execute the first safety procedure or an alternative procedure that, when implemented by the vehicle when the object implements the second safety procedure, is determined to have a lesser likelihood of incurring a collision between the vehicle and the object than the first safety procedure.

Safety procedure analysis for obstacle avoidance in autonomous vehicles

In various examples, one or more output channels of a deep neural network (DNN) may be used to determine assignments of obstacles to paths. To increase the accuracy of the DNN, the input to the DNN may include an input image, one or more representations of path locations, and/or one or more representations of obstacle locations. The system may thus repurpose previously computed information—e.g., obstacle locations, path locations, etc.—from other operations of the system, and use them to generate more detailed inputs for the DNN to increase accuracy of the obstacle to path assignments. Once the output channels are computed using the DNN, computed bounding shapes for the objects may be compared to the outputs to determine the path assignments for each object. Additionally, a machine may perform control operations based at least on the path assignments.

Obstacle to path assignment and performance of control operations based on assignment for autonomous systems and applications

In various examples, live perception from sensors of a vehicle may be leveraged to detect and classify intersection contention areas in an environment of a vehicle in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute outputs—such as signed distance functions—that may correspond to locations of boundaries delineating intersection contention areas. The signed distance functions may be decoded and/or post-processed to determine instance segmentation masks representing locations and classifications of intersection areas or regions. The locations of the intersections areas or regions may be generated in image-space and converted to world-space coordinates to aid an autonomous or semi-autonomous vehicle in navigating intersections according to rules of the road, traffic priority considerations, and/or the like.

Intersection region detection and classification for autonomous machine applications

One or more embodiments of the present disclosure relate to generation of map data. In these or other embodiments, the generation of the map data may include determining whether objects indicated by the sensor data are static objects or dynamic objects. Additionally or alternatively, sensor data may be removed or included in the map data based on determinations as to whether it corresponds to static objects or dynamic objects.

Sensor data based map creation for autonomous systems and applications

In various examples, a three-dimensional (3D) intersection structure may be predicted using a deep neural network (DNN) based on processing two-dimensional (2D) input data. To train the DNN to accurately predict 3D intersection structures from 2D inputs, the DNN may be trained using a first loss function that compares 3D outputs of the DNN—after conversion to 2D space—to 2D ground truth data and a second loss function that analyzes the 3D predictions of the DNN in view of one or more geometric constraints—e.g., geometric knowledge of intersections may be used to penalize predictions of the DNN that do not align with known intersection and/or road structure geometries. As such, live perception of an autonomous or semi-autonomous vehicle may be used by the DNN to detect 3D locations of intersection structures from 2D inputs.

Three-dimensional intersection structure prediction for autonomous driving applications

In various examples, an end-to-end perception evaluation system for autonomous and semi-autonomous machine applications may be implemented to evaluate how the accuracy or precision of outputs of machine learning models—such as deep neural networks (DNNs)—impact downstream performance of the machine when relied upon. For example, decisions computed by the system using ground truth output types may be compared to decisions computed by the system using the perception outputs. As a result, discrepancies in downstream decision making of the system between the ground truth information and the perception information may be evaluated to either aid in updating or retraining of the machine learning model or aid in generating more accurate or precise ground truth information.

End-to-end evaluation of perception systems for autonomous systems and applications

In various examples, object fence corresponding to objects detected by an ego-vehicle may be used to determine overlap of the object fences with lanes on a driving surface. A lane mask may be generated corresponding to the lanes on the driving surface, and the object fences may be compared to the lanes of the lane mask to determine the overlap. Where an object fence is located in more than one lane, a boundary scoring approach may be used to determine a ratio of overlap of the boundary fence, and thus the object, with each of the lanes. The overlap with one or more lanes for each object may be used to determine lane assignments for the objects, and the lane assignments may be used by the ego-vehicle to determine a path or trajectory along the driving surface.

Lane mask generation for autonomous machine

In various examples, systems and methods are disclosed that preserve rich spatial information from an input resolution of a machine learning model to regress on lines in an input image. The machine learning model may be trained to predict, in deployment, distances for each pixel of the input image at an input resolution to a line pixel determined to correspond to a line in the input image. The machine learning model may further be trained to predict angles and label classes of the line. An embedding algorithm may be used to train the machine learning model to predict clusters of line pixels that each correspond to a respective line in the input image. In deployment, the predictions of the machine learning model may be used as an aid for understanding the surrounding environment—e.g., for updating a world model—in a variety of autonomous machine applications.

Regression-based line detection for autonomous driving machines

In various examples, a safety decomposition architecture for autonomous machine applications is presented that uses two or more individual safety assessments to satisfy a higher safety integrity level (e.g., ASIL D). For example, a behavior planner may be used as a primary planning component, and a collision avoidance feature may be used as a diverse safety monitoring component—such that both may redundantly and independently prevent violation of safety goals. In addition, robustness of the system may be improved as single point and systematic failures may be avoided due to the requirement that two independent failures—e.g., of the behavior planner component and the collision avoidance component—occur simultaneously to cause a violation of the safety goals.

Safety decomposition architecture for autonomous machine applications

One or more embodiments of the present disclosure relate to generating RADAR (RAdio Detection And Ranging) point clouds based on RADAR data obtained from one or more RADAR sensors disposed on one or more ego-machines. In these or other embodiments, the RADAR point clouds may be communicated to a distributed map system that is configured to generate map data based on the RADAR point clouds. In some embodiments of the present disclosure, certain compression operations may be performed on the RADAR point clouds to reduce the amount of data that is communicated from the ego-machines to the map system.

Sensor data based map creation and localization for autonomous systems and applications

A trajectory for an autonomous machine may be evaluated for safety based at least on determining whether the autonomous machine would be capable of occupying points of the trajectory in space-time while still being able to avoid a potential future collision with one or more objects in the environment through use of one or more safety procedures. To do so, a point of the trajectory may be evaluated for conflict based at least on a comparison between points in space-time that correspond to the autonomous machine executing the safety procedure(s) from the point and arrival times of the one or more objects to corresponding position(s) in the environment. A trajectory may be sampled and evaluated for conflicts at various points throughout the trajectory. Based on results of one or more evaluations, the trajectory may be scored, eliminated from consideration, or otherwise considered for control of the autonomous machine.

Using arrival times and safety procedures in motion planning trajectories for autonomous vehicles

A deep neural network(s) (DNN) may be used to detect objects from sensor data of a three dimensional (3D) environment. For example, a multi-view perception DNN may include multiple constituent DNNs or stages chained together that sequentially process different views of the 3D environment. An example DNN may include a first stage that performs class segmentation in a first view (e.g., perspective view) and a second stage that performs class segmentation and/or regresses instance geometry in a second view (e.g., top-down). The DNN outputs may be processed to generate 2D and/or 3D bounding boxes and class labels for detected objects in the 3D environment. As such, the techniques described herein may be used to detect and classify animate objects and/or parts of an environment, and these detections and classifications may be provided to an autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle.

Top-down object detection from LiDAR point clouds

Growing community of inventors

Bellevue, WA, United States of America

David Nister