Zoox, Inc.

Peter Scott Schleede

Gary Linscott

Prateek Chandresh Shah

Marin Kobilarov

Ethan Miller Pronovost

Abishek Krishna Akella

Taylor Andrew Arnicar

Joseph H Gribble

William Harland Montgomery, Iv

Ke Sun

Xiangyu Xie

Aaron Huang

Seungji Lee

Samir Joshi

Yan Chang

Shengnan Liang

Kelly Leece Mcguire

Techniques for detecting and/or classifying objects in a driving environment are described herein. A vehicle may receive sensor data and input the sensor data into a perception and/or localization component which may be trained to provide processed output data (e.g., bounding boxes, track(s), etc.). In such cases, the vehicle may generate a scene embedding based on the output data. Further, the vehicle may receive image data from an image capturing device on the vehicle. The vehicle may generate an image embedding associated with the image data and concatenate the image embedding with the scene embedding. The vehicle may input the concatenated embedding into a detection and/or classification model to detect and/or classify object(s) within the environment.

Detection and classification based on scene embeddings and image embeddings

Techniques for generating a tree structure based on multiple machine-learned trajectories are described herein. A planning component ('ML system') within a vehicle may receive and encode various types of sensor and/or vehicle data. The ML system can provide the encoded data as input to multiple machine-learning models (“ML models”), each of which may be trained to output a unique candidate trajectory for the vehicle follow. In some examples, each ML model may be trained to output a unique type of learned trajectory that causes the vehicle to perform a certain type of action. Using the learned candidate trajectories, the ML system may generate a tree structure that includes some or all of the candidate trajectories. The vehicle may determine a control trajectory based on the generation and traversal of the tree structure using a tree search algorithm, and may follow the control trajectory within the environment.

Vehicle trajectory tree structure including learned trajectories

A machine-learned architecture for estimating the cost determined by a cost function for a prediction node of a tree search for exploring potential paths for controlling a vehicle may include two portions: a set up portion that includes models trained to process static data and a second portion that processes dynamic object data. The respective portions of the architecture may comprise various models that determine intermediate outputs that may be projected into a space associated with estimated cost. That estimated cost may identify an estimate of an output of the cost function for paths that are based on a particular prediction node of the tree search.

Machine-learned cost estimation in tree search trajectory generation for vehicle control

A machine-learned architecture for generating a single trajectory or multiple trajectories for controlling a vehicle may comprise an embedding model that generates an embedding of a world state indicating environment, object, and/or other states, one or more machine-learned layers that determine a predicted world state embedding using the world state embedding, and concatenating, as combined data, that predicted world state embedding to a steering angle distribution and a velocity distribution. The combined data is provided as input to a machine-learned model (that may be a single machine-learned model or may comprise two separate machine-learned models) that determines a next steering angle distribution and a next velocity distribution. These distributions may be used as part of generating one or more trajectories for controlling the vehicle. The architecture may be iteratively used to create a series of distributions that are used to create one or more trajectories.

Vehicle control using auto-regressive control distibution generation based on world state embeddings

Techniques are disclosed for utilizing synchronized robot observations. The techniques may include receiving first log data associated with a first vehicle, receiving second log data associated with a second vehicle, determining that a first portion of the first log data is time related to a second portion of the second log data, and using the first portion and the second portion for mapping and/or training models.

Using robot observations

A remote computer system may receive data representative of a state of an autonomous vehicle traversing an environment. A representation of the environment and the state may be input to a first machine learned model, which may output a vector representation of the environment at a first time. The vector representation may be input to a second machine learned model, which may output a second vector representation of the environment at a second time. A third machine learned model may receive the second vector representation and may output a mean velocity and a mean steering angle of the vehicle at the first time and associated uncertainty variables. Based on the mean velocity, the mean steering angle, and the uncertainty variables, a predicted vehicle state at the first time and an associated uncertainty may be determined and the vehicle may be controlled based on the predicted vehicle state and the uncertainty.

Machine-learned model including uncertainty generation

Techniques for determining classifications associated with vehicle behavior by a machine learned model are discussed herein. A computing device can combine a variety of classifier labels as input to train a machine learned model to determine classifications that represent a vehicle behavior as either good behavior or bad behavior. The machine learned model may receive weak classifier labels that classify an aspect of a vehicle behavior as 'good,' “bad,” or “unknown” (relative to the classification) to generate training data for training the machine learning model. Data associated with a vehicle or simulated vehicle may be input to the trained machine learned model to classify good vehicle behavior or bad vehicle behavior even when the model has not been exposed to the precise scenarios represented in the training data.

Training vehicle behavior labels

Techniques for determining whether to yield a vehicle to an oncoming object based on a stationary object are discussed herein. A vehicle can determine a drivable area in an environment through which the vehicle is travelling. A trajectory associated with the vehicle can be determined. A probability that the oncoming object will intersect a drive path associated with a vehicle trajectory can be determined. In some case, the probability can be based on various distance metric(s) of the environment. If the oncoming object is determined to intersect the drive path, a stopping location for the vehicle can be determined to allow the oncoming object to traverse around the stationary object. The vehicle can be controlled based on sensor data captured by the vehicle and the probability.

Collision avoidance for a vehicle with objects in an oncoming lane

Techniques for analyzing driving simulations and determining simulation stability metrics are discussed herein. Various techniques may include receiving a driving simulation and determining a number of perturbed simulations based on the driving simulation. The perturbed simulations may include, for example, one or more perturbed object attributes, perturbed simulation failure conditions, and/or perturbed behaviors of the vehicle system on which the simulation is executed. The perturbed simulations may be executed, and the results may be analyzed to determine a stability score associated with the driving simulation. Stability scores for driving simulations may be determined and maintained based on different simulation data perturbations received based on perturbation models, and the stability scores may be used in various implementations to generate simulation batteries for regression testing and for analyzing and routing simulation failures.

Simulation stability analyses based on parameter perturbation

Techniques for determining whether to yield a vehicle to an oncoming object based on a stationary object are discussed herein. A vehicle can determine a drivable area in an environment through which the vehicle is travelling. A trajectory associated with the vehicle can be determined. A probability that the oncoming object will intersect a drive path associated with a vehicle trajectory can be determined. In some case, the probability can be based on various distance metric(s) (e.g., dynamic distance metrics) of the environment. If the oncoming object is determined to intersect the drive path, a stopping location for the vehicle can be determined to allow the oncoming object to traverse around the stationary object. The vehicle can be controlled based on sensor data captured by the vehicle and the probability.

Dynamic threshold determination for collision avoidance

Techniques relating to performance-based metrics for evaluating system quality are described. In an example, sensor data associated with an environment within which a vehicle is positioned is received. A performance metric associated with a trajectory, indicative of a performed behavior of the vehicle, can be determined and a correctness metric can be determined based at least in part on the performance metric. The correctness metric can represent a correctness of the performed behavior. Modification(s) to component(s) and/or system(s), or portions thereof, of the vehicle can be affected based at least in part on the correctness metric.

Performance-based metrics for evaluating system quality

Techniques for controlling a vehicle based on a collision avoidance algorithm are discussed herein. The vehicle receives sensor data and can determine that the sensor data represents an object in an environment through which the vehicle is travelling. A computing device associated with the vehicle determines a collision probability between the vehicle and the object at predicted locations of the vehicle and object at a first time. Updated locations of the vehicle and object can be determined, and a second collision probability can be determined. The vehicle is controlled based at least in part on the collision probabilities.

Growing community of inventors

Foster City, CA, United States of America

Peter Scott Schleede