Aurora Operations, Inc.

Tandent Vision Science, Inc.

Aurora Innovation, Inc.

Carnegie Mellon University

Google Inc.

Uatc, LLC

James Andrew Bagnell

Ethan Eade

Arun Venkatraman

Sterling J Anderson

Christopher Paul Urmson

Warren Smith

Bartholomeus C Nabbe

Casey Arthur Smith

Andrew Neil Stein

Michael Bode

David Tolliver

Sanjiban Choudhury

Bartholomeus Christianus Leonardus Nabbe

Bruce Allen Maxwell

Richard Mark Friedhoff

Steven Joseph Bushell

Kristin Jean Dana

Timothy King Rodgers, Jr

Christopher Chandler Fromme

Michael Lee Phillips

Herman Herman

David LaRose

Brett Browning

Peter Hansen

Paul Vernaza

Brian C Becker

Robert Zlot

Narek Melik-Barkhudarov

Neil Stegall

David Prasser

Abhay Vardhan

Adam Milstein

Venkatraman Narayanan

Iljung Samuel Kwak

Haoyang Fan

Xinyan Yan

Milan Diebel

Elliot Allen Cuzzillo

Davis Edward King

Michael Knoll Sergi-Curfman

Yekeun Jeong

Adam Richard Williams

Richard Pantaleo

Joan Campoy

Jaime W Bourne

L Douglas Baker

Nicholas George Dilip Roy

David Jonathan Stager

Anind K Dey

Colin Mease

Shervin Javdani

Jason Leu

Juan Pablo Mendoza

Skandavimal Shridhar

Drew Edward Steedly

Jeffrey David McMahill

Brian D Ziebart

David Larose

David Allen Tolliver

Neil Frederick Stegall

Richard D Pantaleo

David Arthur Larose

Lljung Samuel Kwak

Richard Mark Friedhof

Implementations described herein relate to enabling natural language communications with an autonomous vehicle. In some implementations, processor(s) of a system can initiate and conduct a conversation with a remote communication participant that is located remotely from the autonomous vehicle whereas, in additional or alternative implementations, the processor(s) can answer an incoming electronic communication and conduct a conversation with a remote communication participant that is located remotely from the autonomous vehicle. In other additional or alternative implementations, the processor(s) can also conduct conversations with a local communication participant that is located proximate to the autonomous vehicle. Notably, the processor(s) can be implemented locally at the autonomous vehicle or remotely from the autonomous vehicle (e.g., at a remote server).

Natural language communications with an autonomous vehicle

Techniques are disclosed for training one or more cost functions of an autonomous vehicle ('AV') control system based on difference between data generated using the AV control system and manual driving data. In many implementations, manual driving data captures action(s) of a vehicle controlled by a manual driver. Additionally or alternatively, multiple AV control systems can be evaluated by comparing deviations for each AV control system, where the deviations are determined using the same set of manual driving data.

Training machine learning model for controlling autonomous vehicle

Evaluating the performance of an autonomous vehicle includes determining a plurality of simulation scenarios, determining a set of features correlated to a performance metric of interest for the autonomous vehicle, executing a simulation for each simulation scenario in the plurality of simulation scenarios, determining, by a machine learning model, a weight for each simulation scenario in the set of simulation scenarios subject to a constraint that a simulated expected value of each feature in the plurality of simulation scenarios falls within a threshold range of an observed expected value of each feature in an operation design domain of interest of the autonomous vehicle, and estimating an expected value of the performance metric of interest of the autonomous vehicle based on the determined weight and the execution of the simulation for each simulation scenario in the plurality of simulation scenarios.

Estimating autonomous vehicle performance metrics in real world from simulation scenarios

Implementations process, using machine learning (ML) layer(s) of ML model(s), actor(s) from a past episode of locomotion of a vehicle and stream(s) in an environment of the vehicle during the past episode to forecast associated trajectories, for the vehicle and for each of the actor(s), with respect to a respective associated stream of the stream(s). Further, implementations process, using a stream connection function, the associated trajectories to forecast a plurality of associated trajectories, for the vehicle and each of the actor(s), with respect to each of the stream(s). Moreover, implementations iterate between using the ML layer(s) and the stream connection function to update the associated trajectories for the vehicle and each of the actor(s). Implementations subsequently use the ML layer(s) in controlling an AV.

Systems and methods related to controlling autonomous vehicle(s)

Determining an instantaneous vehicle characteristic (e.g., at least one yaw rate) of an additional vehicle that is in addition to a vehicle being autonomously controlled, and adapting autonomous control of the vehicle based on the determined instantaneous vehicle characteristic of the additional vehicle. For example, autonomous steering, acceleration, and/or deceleration of the vehicle can be adapted based on a determined instantaneous vehicle characteristic of the additional vehicle. In many implementations, the instantaneous vehicle characteristics of the additional vehicle are determined based on data from a phase coherent Light Detection and Ranging (LIDAR) component of the vehicle, such as a phase coherent LIDAR monopulse component and/or a frequency-modulated continuous wave (FMCW) LIDAR component.

Control of autonomous vehicle based on determined yaw parameter(s) of additional vehicle

Techniques are disclosed for evaluating manual driving data using an AV control system based on differences between data generated using the AV control system and the manual driving data. In many implementations, manual driving data captures action(s) of a vehicle controlled by a manual driver. Additionally or alternatively, an additional AV control system can be trained using the evaluated manual driving data.

Evaluating driving data using autonomous vehicle control system

The present disclosure provides an example method for validating a trajectory generated by an autonomous vehicle control system (AV trajectory) in a driving scenario. The example method includes (a) obtaining the AV trajectory and a reference trajectory, wherein the reference trajectory describes a desired motion of a vehicle in the driving scenario; (b) determining a plurality of component divergence values for a plurality of divergence metrics, wherein a respective divergence value characterizes a respective difference between the AV trajectory and the reference trajectory; (c) providing the plurality of component divergence values to a machine-learned model to generate a score that indicates an aggregate divergence between the AV trajectory and the reference trajectory, wherein the machine-learned model comprises a plurality of learned parameters defining an influence of the plurality of component divergence values on the score; and (d) validating the AV trajectory based on the score.

Learned validation metric for evaluating autonomous vehicle motion planning performance

An example method includes obtaining log data descriptive of an exemplar action of an exemplar vehicle in an environment, the exemplar action occurring in an initial state of the environment; determining, using the operational system, a planned action for a simulated vehicle in the initial state of the environment; simulating an SUT state of the environment resulting from the simulated vehicle executing the planned action in the initial state of the environment and an actor performing an actor action subsequent to the planned action and an exemplar state of the environment resulting from the simulated vehicle executing the exemplar action in the initial state of the environment and the actor performing the actor action subsequent to the exemplar action; determining a test score based on the SUT state and a reference score based on the exemplar state; evaluating the operational system based on the test score and the reference score.

Systems and methods for autonomous vehicle validation

Systems and methods for training a machine learning ('ML') model for use in controlling an autonomous vehicle (“AV”) are described herein. Implementations can obtain an initial state instance from driving of a vehicle, obtain ground truth label(s) for subsequent state instance(s) that each indicate a corresponding action of the vehicle for a corresponding time instance, perform, for a given time interval, a simulated episode, of locomotion of a simulated AV, generate, for each of a plurality of time instances of the given time interval, subsequent simulated state instance(s) that differ from the subsequent state instance(s), determine, using the ML model, and for each of the time instances, a predicted simulated action of the simulated AV based on the subsequent simulated operation instance(s), generate loss(es) based on the predicted simulated actions and the ground truth labels, and update the ML model based on the loss(es).

Training machine learning model(s), in simulation, for use in controlling autonomous vehicle(s)

Determining classification(s) for object(s) in an environment of autonomous vehicle, and controlling the vehicle based on the determined classification(s). For example, autonomous steering, acceleration, and/or deceleration of the vehicle can be controlled based on determined pose(s) and/or classification(s) for objects in the environment. The control can be based on the pose(s) and/or classification(s) directly, and/or based on movement parameter(s), for the object(s), determined based on the pose(s) and/or classification(s). In many implementations, pose(s) and/or classification(s) of environmental object(s) are determined based on data from a phase coherent Light Detection and Ranging (LIDAR) component of the vehicle, such as a phase coherent LIDAR monopulse component and/or a frequency-modulated continuous wave (FMCW) LIDAR component.

Control of autonomous vehicle based on environmental object classification determined using phase coherent LIDAR data

Techniques are disclosed for evaluating an autonomous vehicle ('AV') control system by determining deviations between data generated using the AV control system and manual driving data. In many implementations, manual driving data captures action(s) of a vehicle controlled by a manual driver. Additionally or alternatively, multiple AV control systems can be evaluated by comparing deviations for each AV control system, where the deviations are determined using the same set of manual driving data.

Evaluating autonomous vehicle control system

Systems and methods for landmark-based localization of an autonomous vehicle ('AV') are described herein. Implementations can generate a first predicted location of a landmark based on a pose instance of a pose of the AV and a stored location of the landmark, generate a second predicted location of the landmark relative to the AV based on an instance of LIDAR data, generate a correction instance based on the comparing, and use the correction instance in generating additional pose instance(s). Systems and methods for validating localization of a vehicle are also described herein. Implementations can obtain driving data from a past episode of locomotion of the vehicle, generate a pose-based predicted location of a landmark in an environment of the vehicle, and compare the pose-based predicted location to a stored location of the landmark in the environment of the vehicle to validate a pose instance of a pose of the vehicle.

Landmark-based localization methods and architectures for an autonomous vehicle

Example methods for multistage autonomous vehicle motion planning include obtaining sensor data descriptive of an environment of the autonomous vehicle; identifying one or more objects in the environment based on the sensor data; generating a plurality of candidate strategies, wherein each candidate strategy of the plurality of candidate strategies comprises a set of discrete decisions respecting the one or more objects, wherein generating the plurality of candidate strategies includes: determining that at least two strategies satisfy an equivalence criterion, such that the plurality of candidate strategies include at least one candidate strategy corresponding to an equivalence class representative of a plurality of different strategies that are based on different discrete decisions; determining candidate trajectories respectively for the plurality of candidate strategies; and initiating control of the autonomous vehicle based on a selected candidate trajectory.

Multistage autonomous vehicle motion planning

Growing community of inventors

Pittsburgh, PA, United States of America

James Andrew Bagnell