Nvidia Corporation

Leland Stanford Junior University

Toyota Research Institute, Inc.

Ford Global Technolgoies, LLC

Marco Pavone

Boris Ivanovic

Adrien David Gaidon

Sushant Veer

Karen Yan Ming Leung

Guy Rosman

Punarjay Chakravarty

Shie Mannor

Shubham Shrivastava

Nikos Arechiga Gonzalez

Mac Schwager

Haruki Nishimura

Karen Y Leung

Peter Karkus

Yuxiao Chen

Yifeng Lin

Amine Elhafsi

Alec Farid

Edward Fu Schmerling

In various examples, control policies for controlling agents may be learned from demonstrations capturing joint states of entities navigating through the environment. A control policy may be learned mapping joint states to control actions, where the joint states are between agents, and the control actions are of at least one of the agents. The control policy may be learned to define the mappings as control invariant sets of the joint sates and the control actions. The control policy may be used to determine one or more functions that compute, based at least on a joint state between entities, output indicating a likelihood of collision between the entities operating in accordance with the control policy. Using the output, current and/or potential states of the environment may be evaluated to determine control operations for a machine, such as a vehicle.

Learning autonomous vehicle safety concepts from demonstrations

In various examples, cost probability distributions corresponding to predicted locations of an object in an environment and potential locations for a machine in the environment and may be evaluated using corresponding observed costs corresponding to the machine and the object. The cost probability distributions may be evaluated based on comparing the observed costs to threshold values, which may be determined based on sampling a predicted cost function. A threshold value may be selected to provide false-positive rate and/or false-negative rate guarantees for anomaly detection. Control operations may be performed based on results of the evaluation of the cost probability distributions. For example, based on the results, a motion planner may reuse a planned trajectory for a future planning cycle (e.g., thereby avoiding re-planning computations) or generate and/or select a new planned trajectory (e.g., based at least on one or more anomalies being detected).

Task-relevant failure detection for trajectory prediction in machines

Apparatuses, systems, and techniques to generate trajectory predictions. In at least one embodiment, trajectory predictions are generated based on, for example, one or more neural networks.

Neural network trajectory prediction

In various examples, a motion planner include an analytical function to predict motion plans for a machine based on predicted trajectories of actors in an environment, where the predictions are differentiable with respect to parameters of a neural network of a motion predictor used to predict the trajectories. The analytical function may be used to determine candidate trajectories for the machine based on a predicted trajectory, to compute cost values for the candidate trajectories, and to select a reference trajectory from the candidate trajectories. For differentiability, a term of the analytical function may correspond to the predicted trajectory. A motion controller may use the reference trajectory to predict a control sequence for the machine using an analytical function trained to generate predictions that are differentiable with respect to at least one parameter of the analytical function used to compute the cost values.

Differentiable and modular prediction and planning for autonomous machines

A mobile robot can be caused to move according to a planned trajectory. The mobile robot can be a vehicle. Information about agents in an environment of the mobile robot can be received from sensors. At a first time, a spatiotemporal graph can be produced. The spatiotemporal graph can represent relationships among the agents in the environment. The mobile robot can be one of the agents in the environment. Information from the spatiotemporal graph can be input to neural networks to produce information for a mixture of affine time-varying systems. The mixture of affine time-varying systems can represent an evolution of agent states of the agents. Using the mixture of affine time-varying systems and information associated with the first time, a prediction of the agent states at a second time can be calculated. The mobile robot can be caused to move according to the planned trajectory determined from the prediction.

Causing a mobile robot to move according to a planned trajectory determined from a prediction of agent states of agents in an environment of the mobile robot

A plurality of agent locations can be determined at a plurality of time steps by inputting a plurality of images to a perception algorithm that inputs the plurality of images and outputs agent labels and the agent locations. A plurality of first uncertainties corresponding to the agent locations can be determined at the plurality of time steps by inputting the plurality of agent locations to a filter algorithm that inputs the agent locations and outputs the plurality of first uncertainties corresponding to the plurality of agent locations. A plurality of predicted agent trajectories and potential trajectories corresponding to the predicted agent trajectories can be determined by inputting the plurality of agent locations at the plurality of time steps and the first uncertainties corresponding to the agent locations at the plurality of time steps to a variational autoencoder. The plurality of predicted agent trajectories and the potential trajectories corresponding to the predicted agent trajectories can be output.

Object trajectory forecasting

Systems and methods for generating and evaluating driving scenarios with varying difficulty levels is provided. The disclosed systems and methods may be used to develop a suite of regression tests that track the progress of an autonomous driving stack. A robustness trace of a temporal logic formula may be computed from an always-eventually fragment using a computation graph. The robustness trace may be approximated by a smoothly differentiable computation graph, which can be implemented in existing machine learning programming frameworks. The systems and methods provided herein may be useful in automatic test case generation for autonomous or semi-autonomous vehicles.

Systems and methods for automatic test generation

Systems, methods, and other embodiments described herein relate to improving controls in a device according to risk. In one embodiment, a method includes, in response to receiving sensor data about a surrounding environment of the device, identifying objects from the sensor data that are present in the surrounding environment. The method includes generating a control sequence for controlling the device according to a risk-sensitivity parameter to navigate toward a destination while considering risk associated with encountering the objects defined by the risk-sensitivity parameter. The method includes controlling the device according to the control sequence.

Growing community of inventors

Stanford, CA, United States of America

Marco Pavone