Nvidia Corporation

Ricoh Company, Ltd.

Junghyun Kwon

Minwoo Park

David Nister

Sangmin Oh

Berta Rodriguez Hervas

Hae-Jong Seo

Mehmet K Kocamaz

Neda Cvijetic

Dongwoo Lee

Max E McFarland

Silvio Savarese

Zhaoting Ye

Yilin Yang

Trung Pham

Bala Siva Sashank Jujjavarapu

Tae Eun Choe

Hang Dou

Igor Tryndin

Wenchao Zheng

Sayed Mehdi Sajjadi Mohammadabadi

Edward L Schwartz

Kathrin Berkner

Ramya Narasimha

Jagadeesh Sankaran

Wanli Jiang

Dong Zhang

Eric Viscito

Baris Evrim Demiroz

Yichun Shen

Siyi Li

Ke Xu

Dinghuang Ji

Chethan Ningaraju

Hao Tsui

Yongqing Liang

Neeraj Sajjan

Max Mcfarland

In various examples, live perception from sensors of a vehicle may be leveraged to detect and classify intersections in an environment of a vehicle in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute various outputs—such as bounding box coordinates for intersections, intersection coverage maps corresponding to the bounding boxes, intersection attributes, distances to intersections, and/or distance coverage maps associated with the intersections. The outputs may be decoded and/or post-processed to determine final locations of, distances to, and/or attributes of the detected intersections.

Intersection detection and classification in autonomous machine applications

A neural network may be used to determine corner points of a skewed polygon (e.g., as displacement values to anchor box corner points) that accurately delineate a region in an image that defines a parking space. Further, the neural network may output confidence values predicting likelihoods that corner points of an anchor box correspond to an entrance to the parking spot. The confidence values may be used to select a subset of the corner points of the anchor box and/or skewed polygon in order to define the entrance to the parking spot. A minimum aggregate distance between corner points of a skewed polygon predicted using the CNN(s) and ground truth corner points of a parking spot may be used simplify a determination as to whether an anchor box should be used as a positive sample for training.

Object detection using skewed polygons suitable for parking space detection

In various examples, one or more output channels of a deep neural network (DNN) may be used to determine assignments of obstacles to paths. To increase the accuracy of the DNN, the input to the DNN may include an input image, one or more representations of path locations, and/or one or more representations of obstacle locations. The system may thus repurpose previously computed information—e.g., obstacle locations, path locations, etc.—from other operations of the system, and use them to generate more detailed inputs for the DNN to increase accuracy of the obstacle to path assignments. Once the output channels are computed using the DNN, computed bounding shapes for the objects may be compared to the outputs to determine the path assignments for each object. Additionally, a machine may perform control operations based at least on the path assignments.

Obstacle to path assignment and performance of control operations based on assignment for autonomous systems and applications

In various examples, a multi-sensor fusion machine learning model—such as a deep neural network (DNN)—may be deployed to fuse data from a plurality of individual machine learning models. As such, the multi-sensor fusion network may use outputs from a plurality of machine learning models as input to generate a fused output that represents data from fields of view or sensory fields of each of the sensors supplying the machine learning models, while accounting for learned associations between boundary or overlap regions of the various fields of view of the source sensors. In this way, the fused output may be less likely to include duplicate, inaccurate, or noisy data with respect to objects or features in the environment, as the fusion network may be trained to account for multiple instances of a same object appearing in different input representations.

Sensor fusion for autonomous machine applications using machine learning

In various examples, live perception from sensors of a vehicle may be leveraged to generate object tracking paths for the vehicle to facilitate navigational controls in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute various outputs—such as feature descriptor maps including feature descriptor vectors corresponding to objects included in a sensor(s) field of view. The outputs may be decoded and/or otherwise post-processed to reconstruct object tracking and to determine proposed or potential paths for navigating the vehicle.

Single and across sensor object tracking using feature descriptor mapping in autonomous systems and applications

Apparatuses, systems, and techniques to perform non-maximum suppression (NMS) in parallel to remove redundant bounding boxes. In at least one embodiment, two or more parallel circuits to perform two or more portions of a NMS algorithm in parallel to remove one or more redundant bounding boxes corresponding to one or more objects within one or more digital images.

Performing non-maximum suppression in parallel

In various examples, a deep neural network (DNN) is trained—using image data alone—to accurately predict distances to objects, obstacles, and/or a detected free-space boundary. The DNN may be trained with ground truth data that is generated using sensor data representative of motion of an ego-vehicle and/or sensor data from any number of depth predicting sensors—such as, without limitation, RADAR sensors, LIDAR sensors, and/or SONAR sensors. The DNN may be trained using two or more loss functions each corresponding to a particular portion of the environment that depth is predicted for, such that—in deployment—more accurate depth estimates for objects, obstacles, and/or the detected free-space boundary are computed by the DNN.

Distance estimation to objects and free-space boundaries in autonomous machine applications

In various examples, live perception from sensors of an ego-machine may be leveraged to detect objects and assign the objects to bounded regions (e.g., lanes or a roadway) in an environment of the ego-machine in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute outputs—such as output segmentation masks—that may correspond to a combination of object classification and lane identifiers. The output masks may be post-processed to determine object to lane assignments that assign detected objects to lanes in order to aid an autonomous or semi-autonomous machine in a surrounding environment.

Assigning obstacles to lanes using neural networks for autonomous machine applications

A geometric approach may be used to detect objects on a road surface. A set of points within a region of interest between a first frame and a second frame are captured and tracked to determine a difference in location between the set of points in two frames. The first frame may be aligned with the second frame and the first pixel values of the first frame may be compared with the second pixel values of the second frame to generate a disparity image including third pixels. One or more subsets of the third pixels that have a value above a first threshold may be combined, and the third pixels may be scored and associated with disparity values for each pixel of the one or more subsets of the third pixels. A bounding shape may be generated based on the scoring.

Object detection using image alignment for autonomous machine applications

In various examples, a deep neural network (DNN) is trained&#x2014;using image data alone&#x2014;to accurately predict distances to objects, obstacles, and/or a detected free-space boundary. The DNN may be trained with ground truth data that is generated using sensor data representative of motion of an ego-vehicle and/or sensor data from any number of depth predicting sensors&#x2014;such as, without limitation, RADAR sensors, LIDAR sensors, and/or SONAR sensors. The DNN may be trained using two or more loss functions each corresponding to a particular portion of the environment that depth is predicted for, such that&#x2014;in deployment&#x2014;more accurate depth estimates for objects, obstacles, and/or the detected free-space boundary are computed by the DNN. In some embodiments, a sampling algorithm may be used to sample depth values corresponding to an input resolution of the DNN from a predicted depth map of the DNN at an output resolution of the DNN.

In various examples, a multi-sensor fusion machine learning model&#x2014;such as a deep neural network (DNN)&#x2014;may be deployed to fuse data from a plurality of individual machine learning models. As such, the multi-sensor fusion network may use outputs from a plurality of machine learning models as input to generate a fused output that represents data from fields of view or sensory fields of each of the sensors supplying the machine learning models, while accounting for learned associations between boundary or overlap regions of the various fields of view of the source sensors. In this way, the fused output may be less likely to include duplicate, inaccurate, or noisy data with respect to objects or features in the environment, as the fusion network may be trained to account for multiple instances of a same object appearing in different input representations.

In various examples, live perception from sensors of a vehicle may be leveraged to detect and classify intersections in an environment of a vehicle in real-time or near real-time. For example, a deep neural network (DNN) may be trained to compute various outputs&#x2014;such as bounding box coordinates for intersections, intersection coverage maps corresponding to the bounding boxes, intersection attributes, distances to intersections, and/or distance coverage maps associated with the intersections. The outputs may be decoded and/or post-processed to determine final locations of, distances to, and/or attributes of the detected intersections.

Growing community of inventors

San Jose, CA, United States of America

Junghyun Kwon