Nec Corporation

Aurora Operations, Inc.

Paul Vernaza

Manmohan Chandraker

Samuel Schulter

Wongun Choi

Quoc-Huy Tran

Nicholas Rhinehart

Muhammad Zeeshan Zia

Mohammed E F Salem

Menghua Zhai

James Andrew Bagnell

Kihyuk Sohn

Yi-Hsuan Tsai

Buyu Liu

Arun Venkatraman

Pan Ji

Pengju Jin

Namhoon Lee

Brian D Ziebart

Anqi Liu

Juan Pablo Mendoza

Mohammed E Fathy Salem

Evaluating the performance of an autonomous vehicle includes determining a plurality of simulation scenarios, determining a set of features correlated to a performance metric of interest for the autonomous vehicle, executing a simulation for each simulation scenario in the plurality of simulation scenarios, determining, by a machine learning model, a weight for each simulation scenario in the set of simulation scenarios subject to a constraint that a simulated expected value of each feature in the plurality of simulation scenarios falls within a threshold range of an observed expected value of each feature in an operation design domain of interest of the autonomous vehicle, and estimating an expected value of the performance metric of interest of the autonomous vehicle based on the determined weight and the execution of the simulation for each simulation scenario in the plurality of simulation scenarios.

Estimating autonomous vehicle performance metrics in real world from simulation scenarios

An example method for controlling an autonomous vehicle includes generating, using perception data, a dynamic cost profile associated with an object in the environment; obtaining map data describing lanes of the multilane roadway; generating, using the map data, a lane cost profile that includes a plurality of basins respectively associated with the lanes; generating a candidate trajectory for the autonomous vehicle to traverse in the environment; evaluating the candidate trajectory using an aggregate cost of the candidate trajectory, the aggregate cost computed using the lane cost profile and the dynamic cost profile; determining a selected trajectory for execution by the autonomous vehicle based on the evaluation of the candidate trajectory; controlling the autonomous vehicle according to the selected trajectory.

Lane change architecture for autonomous vehicles

Techniques for improving the performance of an autonomous vehicle (AV) are described herein. A system can determine a plan for the AV in a driving scenario that optimizes an initial cost function of a control algorithm of the AV. The system can obtain data describing an observed human driving path in the driving scenario. Additionally, the system can determine for each cost dimension in the plurality of cost dimensions, a quantity that compares the estimated cost to the observed cost of the observed human driving path. Moreover, the system can determine a function of a sum of the quantities determined for each cost dimension in the plurality of cost dimensions. Subsequently, the system can use an optimization algorithm to adjust one or more weights of the plurality of weights applied to the plurality of cost dimensions to optimize the function of the sum of the quantities.

Systems and methods for pareto domination-based learning

A method is provided in an Advanced Driver-Assistance System (ADAS). The method extracts, from an input video stream including a plurality of images using a multi-task Convolutional Neural Network (CNN), shared features across different perception tasks. The perception tasks include object detection and other perception tasks. The method concurrently solves, using the multi-task CNN, the different perception tasks in a single pass by concurrently processing corresponding ones of the shared features by respective different branches of the multi-task CNN to provide a plurality of different perception task outputs. Each respective different branch corresponds to a respective one of the different perception tasks. The method forms a parametric representation of a driving scene as at least one top-view map responsive to the plurality of different perception task outputs. The method controls an operation of the vehicle for collision avoidance responsive to the at least one top-view map indicating an impending collision.

Multi-task perception network with applications to scene understanding and advanced driver-assistance system

Systems and methods for vehicle behavior prediction include an imaging device that captures images of a vehicle in traffic. A processing device including policy stored in a memory of the processing device in communication with the imaging device stochastically models future behavior of the vehicle based on the captured images. A policy simulator in communication with the processing device simulates the policy as a reparameterized pushforward policy of a base distribution. An evaluator receives the simulated policy from the policy simulator and performs cross-entropy optimization on the future behavior of the vehicle by analyzing the simulated policy and updating the policy according to cross-entropy error. An alert system retrieves the future behavior of the vehicle and recognizes hazardous trajectories of the future trajectories and generates an audible alert using a speaker.

Traffic prediction with reparameterized pushforward policy for autonomous vehicles

Systems and methods for training and evaluating a deep generative model with an architecture consisting of two complementary density estimators are provided. The method includes receiving a probabilistic model of vehicle motion, and training, by a processing device, a first density estimator and a second density estimator jointly based on the probabilistic model of vehicle motion. The first density estimator determines a distribution of outcomes and the second density estimator estimates sample quality. The method also includes identifying by the second density estimator spurious modes in the probabilistic model of vehicle motion. The probabilistic model of vehicle motion is adjusted to eliminate the spurious modes.

Balancing diversity and precision of generative models with complementary density estimators

A method for estimating dense 3D geometric correspondences between two input point clouds by employing a 3D convolutional neural network (CNN) architecture is presented. The method includes, during a training phase, transforming the two input point clouds into truncated distance function voxel grid representations, feeding the truncated distance function voxel grid representations into individual feature extraction layers with tied weights, extracting low-level features from a first feature extraction layer, extracting high-level features from a second feature extraction layer, normalizing the extracted low-level features and high-level features, and applying deep supervision of multiple contrastive losses and multiple hard negative mining modules at the first and second feature extraction layers. The method further includes, during a testing phase, employing the high-level features capturing high-level semantic information to obtain coarse matching locations, and refining the coarse matching locations with the low-level features to capture low-level geometric information for estimating precise matching locations.

Dense three-dimensional correspondence estimation with multi-level metric learning and hierarchical matching

Systems and methods for detecting traffic scenarios include an image capturing device which captures two or more images of an area of a traffic environment with each image having a different view of vehicles and a road in the traffic environment. A hierarchical feature extractor concurrently extracts features at multiple neural network layers from each of the images, with the features including geometric features and semantic features, and for estimating correspondences between semantic features for each of the images and refining the estimated correspondences with correspondences between the geometric features of each of the images to generate refined correspondence estimates. A traffic localization module uses the refined correspondence estimates to determine locations of vehicles in the environment in three dimensions to automatically determine a traffic scenario according to the locations of vehicles. A notification device generates a notification of the traffic scenario.

Computer aided traffic enforcement using dense correspondence estimation with multi-level metric learning and hierarchical matching

Systems and methods for predicting vehicle behavior includes capturing images of a vehicle in traffic using an imaging device. Future behavior of the vehicle is stochastically modeled using a processing device including an energy-based model stored in a memory of the processing device. The energy-based model includes generating a distribution of possible future trajectories of the vehicle using a generator, sampling the distribution of possible future trajectories according to an energy value of each trajectory in the distribution of possible future trajectories an energy model to determine probable future trajectories, and optimizing parameters of each of the generator and the energy model using an optimizer. A user is audibly alerted with a speaker upon an alert system recognizing hazardous trajectories of the probable future trajectories.

Generative adversarial inverse trajectory optimization for probabilistic vehicle forecasting

Systems and methods for correspondence estimation and flexible ground modeling include communicating two-dimensional (2D) images of an environment to a correspondence estimation module, including a first image and a second image captured by an image capturing device. First features, including geometric features and semantic features, are hierarchically extract from the first image with a first convolutional neural network (CNN) according to activation map weights, and second features, including geometric features and semantic features, are hierarchically extracted from the second image with a second CNN according to the activation map weights. Correspondences between the first features and the second features are estimated, including hierarchical fusing of geometric correspondences and semantic correspondences. A 3-dimensional (3D) model of a terrain is estimated using the estimated correspondences belonging to the terrain surface. Relative locations of elements and objects in the environment are determined according to the 3D model of the terrain. A user is notified of the relative locations.

Dense correspondence estimation with multi-level metric learning and hierarchical matching

Systems and methods for generating an occlusion-aware bird's eye view map of a road scene include identifying foreground objects and background objects in an input image to extract foreground features and background features corresponding to the foreground objects and the background objects, respectively. The foreground objects are masked from the input image with a mask. Occluded objects and depths of the occluded objects are inferred by predicting semantic features and depths in masked areas of the masked image according to contextual information related to the background features visible in the masked image. The foreground objects and the background objects are mapped to a three-dimensional space according to locations of each of the foreground objects, the background objects and occluded objects using the inferred depths. A bird's eye view is generated from the three-dimensional space and displayed with a display device.

Generating occlusion-aware bird eye view representations of complex road scenes

Methods and systems for predicting a trajectory include determining prediction samples for agents in a scene based on a past trajectory. The prediction samples are ranked according to a likelihood score that incorporates interactions between agents and semantic scene context. The prediction samples are iteratively refined using a regression function that accumulates scene context and agent interactions across iterations. A response activity is triggered when the prediction samples satisfy a predetermined condition.

Dynamic scene prediction with multiple interacting agents

A surveillance system and method are provided. The surveillance system includes at least one camera configured to capture a set of images of a given target area that includes a set of objects to be tracked. The surveillance system includes a memory storing a learning model configured to perform multi-object tracking by jointly learning arbitrarily parameterized and differentiable cost functions for all variables in a linear program that associates object detections with bounding boxes to form trajectories. The surveillance system includes a processor configured to perform surveillance of the target area to (i) detect the objects and track locations of the objects by applying the learning model to the images in a surveillance task that uses the multi-object tracking, and (ii), provide a listing of the objects and their locations for surveillance task. A bi-level optimization is used to minimize a loss defined on a solution of the linear program.

Growing community of inventors

Sunnyvale, CA, United States of America

Paul Vernaza