Zoox, Inc.

Andrew Scott Crego

Siavosh Rezvan Behbahani

Andreas Christian Reschka

Sai Anurag Modalavalasa

Gerrit Bagschik

James William Vaisey Philbin

Ali Ghasemzadehkhoshgroudi

Lingqiao Qin

Antoine Ghislain Deux

Rodin Lyasoff

Kai Zhenyu Wang

Ashutosh Gajanan Rege

Gowtham Garimella

Marc Wimmershoff

Aditya Pramod Khadilkar

Subhasis Das

Chuang Wang

Josh Alexander Jimenez

Mahsa Ghafarianzadeh

Andraz Kavalar

Michael Haggblade

Muhammad Farooq Rama

Rasmus Fonseca

Kshitij Agarwal

Gonzalo Javier Rey

Techniques and methods for performing collision monitoring using error models. For instance, a vehicle may generate sensor data using one or more sensors. The vehicle may then analyze the sensor data using systems in order to determine parameters associated with the vehicle and parameters associated with another object. Additionally, the vehicle may process the parameters associated with the vehicle using error models associated with the systems in order to determine a distribution of estimated locations associated with the vehicle. The vehicle may also process the parameters associated with the object using the error models in order to determine a distribution of estimated locations associated with the object. Using the distributions of estimated locations, the vehicle may determine the probability of collision between the vehicle and the object.

Collision monitoring using statistic models

An error estimation component may generate contours or a heat map of a likelihood that an object detection or prediction generated by a perception component of an autonomous vehicle is inaccurate. The contours or heat map may be used to lock or unlock actions available to a planning component and the planning component may use the contours or heat for determining a trajectory for controlling the autonomous vehicle. The planning component may additionally or alternatively alter operation of a sensor or determine a trajectory to unlock a target action that is currently locked due to a size/extent of uncertainty indicated by a contour/heat map.

Perception error model for fast simulation and estimation of perception system reliability and/or for control system tuning

Techniques for receiving and processing sensor data captured by a fleet of vehicle are discussed herein. In some examples, a fleet dashcam system can receive sensor data captured by electronic devices on a fleet of vehicles and can use that data to detect collision and near-collision events. The data of the collision or near-collision event can be used to determine a simulation scenario and a response of an autonomous vehicle control to the simulation scenario and/or it can be used to create a collision heat map to aid in operation of an autonomous vehicle.

Fleet dashcam system for autonomous vehicle operation

Techniques for top-down scene discrimination are discussed. A system receives scene data associated with an environment proximate a vehicle. The scene data is input to a convolutional neural network (CNN) discriminator trained using a generator and a classification of the output of the CNN discriminator. The CNN discriminator generates an indication of whether the scene data is a generated scene or a captured scene. If the scene data is data generated scene, the system generates a caution notification indicating that a current environmental situation is different from any previous situations. Additionally, the caution notification is communicated to at least one of a vehicle system or a remote vehicle monitoring system.

Top-down scene discrimination

Techniques for determining a safety margin by which to limit trajectory(ies) generated by a vehicle control system such that the vehicle will not exceed the safety margin more than a target occurrence rate. The techniques may include determining a first spectrum associated with trajectory data generated by one or more vehicles, generating a model of a vehicle, and determining a spectrum of an error signal based at least in part on the model and the first spectrum. Determining the safety margin may be based at least in part on the spectrum of the error signal and a target occurrence rate. Operation characteristics of components of the vehicle (e.g., controller, steering actuator) may be tuned based at least in part on the model, first spectrum, and/or second spectrum. The techniques enable determining safety margins for untested vehicles and/or for different operating states of a vehicle.

Vehicle trajectory controller safety margin

Techniques for modeling the probability distribution of errors in perception systems are discussed herein. For example, techniques may include modeling error distribution for attributes such as position, size, pose, and velocity of objects detected in an environment, and training a mixture model to output specific error probability distributions based on input features such as object classification, distance to the object, and occlusion. The output of the trained model may be used to control the operation of a vehicle in an environment, generate simulations, perform collision probability analyses, and to mine log data to detect collision risks.

Perception error modeling

Techniques for top-down scene generation are discussed. A generator component may receive multi-dimensional input data associated with an environment. The generator component may generate, based at least in part on the multi-dimensional input data, a generated top-down scene. A discriminator component receives the generated top-down scene and a real top-down scene. The discriminator component generates binary classification data indicating whether an individual scene in the scene data is classified as generated or classified as real. The binary classification data is provided as a loss to the generator component and the discriminator component.

Top-down scene generation

Fast simulation of a scenario (e.g., simulating the scenario once as opposed to multiple times) to determine performance metric(s) of a configuration of one or more components of an autonomous vehicle may include training a perception error model based at least in part on a difference between a prediction output by a perception component associated with a future time and a perception output associated with that future time once that future time has arrived. A contour or heat map output by the perception error model may be used to determine one or more performance metric(s) associated with a component of the autonomous vehicle and identify which component may cause a degradation of a performance metric.

Techniques for determining an error model based on vehicle data and ground truth data are discussed herein. To determine whether a complex system (which may be not capable of being inspected) is able to operate safely, various operating regimes (scenarios) can be identified based on operating data. To provide safe operation of such a system, an error model can be determined that can provide a probability associated with perception data and a vehicle can determine a trajectory based on the probability of an error associated with the perception data.

Perception error models

Techniques and methods for performing collision monitoring using system data. For instance, a vehicle may generate sensor data using one or more sensors. The vehicle may then analyze the sensor data using systems in order to determine parameters associated with the vehicle and parameters associated with another object. Additionally, the vehicle may determine uncertainties associated with the parameters and then process the parameters using the uncertainties. Based at least in part on the processing, the vehicle may determine a distribution of estimated locations associated with the vehicle and a distribution of estimated locations associated with the object. Using the distributions of estimated locations, the vehicle may determine the probability of collision between the vehicle and the object.

Collision monitoring using system data

Techniques for predicting safety metrics associated with near-miss conditions for a vehicle, such as an autonomous vehicle, are discussed herein. For instance, a training system identifies an object in an environment and determines a trajectory for the object. The training system may receive a trajectory for a vehicle and associate the trajectory for the object and the trajectory for the vehicle with an event involving the object and the vehicle. In examples, the training system determines a parameter associated with motion of the vehicle as indicated by the trajectory of the vehicle relative to the trajectory of the object, and the event. Then, the training system may determine a safety metric associated with the event that indicates whether the vehicle came within a threshold of a collision with the object during a time period associated with the event.

Safety metric prediction

Techniques associated with generating simulation scenarios for simulating a vehicle controller are discussed herein. Log data may include sensor data captured by sensors of a vehicle. The log data may represent objects in an environment. Objects may be associated with a region of a discretized representation of the environment relative to the vehicle. Specific states of objects (relative position in a region type, velocity, classification, size, etc.) may represent an instance of an occupation. Log data can be aggregated based on similar region type and/or object state. A statistical model over object states can be determined for each region type and can later be sampled to determine simulation parameters. A simulation scenario can be generated based on the simulation parameters, and a vehicle controller can be evaluated based on the simulation scenario.

System for generating generalized simulation scenarios

Techniques for determining a safety metric associated with a vehicle controller are discussed herein. To determine whether a complex system (which may be uninspectable) is able to operate safely, various operating regimes (scenarios) can be identified based on operating data and associated with a scenario parameter to be adjusted. To validate safe operation of such a system, a scenario may be identified for inspection. Error metrics of a subsystem of the system can be quantified. The error metrics, in addition to stochastic errors of other systems/subsystems can be introduced to the scenario. The scenario parameter may also be perturbed. Any multitude of such perturbations can be instantiated in a simulation to test, for example, a vehicle controller. A safety metric associated with the vehicle controller can be determined based on the simulation, as well as causes for any failures.

Safety analysis framework

Techniques and methods for training and/or using a machine learned model that identifies unsafe events. For instance, computing device(s) may receive input data, such as vehicle data generated by one or more vehicles and/or simulation data representing a simulated environment. The computing device(s) may then analyze features represented by the input data using one or more criteria in order to identify potential unsafe events represented by the input data. Additionally, the computing device(s) may receive ground truth data classifying the identified events as unsafe events or safe events. The computing device(s) may then train the machine learned model using at least the input data representing the unsafe events and the classifications. Next, when the computing device(s) and/or vehicles receive input data, the computing device(s) and/or vehicles may use the machine learned model to determine if the input data represents unsafe events.

Growing community of inventors

Foster City, CA, United States of America

Andrew Scott Crego