Zoox, Inc.

Francesco Papi

Subhasis Das

Michael Carsten Bosse

Qian Song

Zachary Sun

Adrian Michael Costantino

John Bryan Carter

Shuangting Liu

Brice Rebsamen

David Burdick Berman

Guillermo Duenas Arana

Gerry Chen

Chuang Wang

Derek Adams

Daniel Miller

Allan Zelener

Stanley L Volta

Shiwei Sheng

Richard Martin Murray

Jifei Qian

Soroush Dean Khadem

Shida Shen

Po-Jen Lai

Anton Mario Bongio Karrman

Anne-Claire Elisabeth Marie Le Henaff

Nicholas George Dilip Roy

Anne-Claire Elisabeth Marie Le Hénaff

Yunming Shao

Cong Ding

Murat Gevrekci

A modified Kalman filter may include one or more neural networks to augment or replace components of the Kalman filter in such a way that the human interpretability of the filter's inner functions is preserved. The neural networks may include a neural network to account for bias in measurement data, a neural network to account for unknown controls in predicting a state of an object, a neural network ensemble that is trained differently based on different sensor data, a neural network for determining the Kalman gain, and/or a set of Kalman filters including various neural networks that determine independent estimated states, which may be fused using Bayesian fusion to determine a final estimated state.

Interpretable Kalman filter comprising neural network component(s) for autonomous vehicles

Systems and techniques for determining more accurate object parameter values, such as orientation values (e.g., yaw, location, position, etc.), for objects detected in an environment are disclosed. A vehicle computing system may receive sensor data from multiple sensor systems that indicates a parameter value determined at the individual sensor systems. The vehicle computing system may determine values and probabilities for particular parameters modes based on the sensor data parameter values and may further filter these values using a mixture model to determine probability distributions for the modes and associated values. These filtered values and modes may then be used to determine predicted object trajectories that can be used to control a vehicle.

Determining object orientation based on parameter modes

Object detection and tracking systems may use machine-learned transformer models with self-attention for detecting, classifying, and/or tracking objects in an environment. Techniques described herein may include receiving sensor data generated by different sensor modalities of a vehicle, determining different bounding shapes based on the different sensor modalities, and using a machine-learned transformer model to determine associated and/or combined bounding shapes. The machine-learned transformer model may receive a variable number of input bounding shapes representing any number of objects and various sensor modalities. Multiple stages of the transformer may be used to determine associated bounding shapes and to assign attributes for the associated bounding shapes, based on the individual bounding shapes of the different sensor modalities and/or previous bounding shapes for objects detected and tracked in a previous scene in the environment.

Object detection and tracking using machine learning transformer models with attention

Techniques for performing object detection in a vehicle environment using sensor data captured by one or more sensors of the vehicle are described herein. In some cases, an object in a vehicle environment can be detected based on at least one of (i) a first similarity matrix that represents a first similarity value for two sensor observations associated with the vehicle environment, or (ii) a second similarity matrix that represents a second similarity value for a sensor observation and a track associated with the vehicle environment.

Object detection using similarity determinations for sensor observations

A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Articulated object determination

A computer-implemented method. Includes determining an estimate of a state of an object detected at a first time step, based on a pointwise estimate of the state of the object at the first time step and pointwise measurements of the state of the object at a plurality of further time steps. Includes generating, using the estimate of the state of the object, a proposed annotation associated with the object at the first time step. Includes rendering, via a user interface, a visual representation of the environment at the first time step and a visual representation of the proposed annotation. Includes receiving, via the user interface, user input indicating a user-approved annotation associated with the object at the first time step. Includes generating training data for a machine learning model for use in controlling an autonomous vehicle, based at least in part on the user-approved annotation.

Assisted labelling of training data for machine learning models

Techniques for improving training data used to train a model to detect objects in an environment are disclosed. Portions of a dataset that are associated with significant operational conditions may be associated with indications of such conditions. The properties of these portions may be used to identify other portions of the dataset or other datasets for augmentation. Data in these portions of a dataset representing detections in an environment may be augmented generally, for example, by flipping geometric values and modifying associated data accordingly. Specific objects represented in such portions may also be augmented by determining particular objects that are represented in selected frames and channels and modifying the pixel values for pixels associated with such objects. The pixel values may be modified or zeroed based on various criteria. The resulting augmented data may be used to train a machine-learned model to more accurately detect objects in an environment.

Object detection using augmented data

An evaluation computing system may implement techniques to validate a vehicle controller, such as based on an update thereto. The evaluation computing system may access data associated with an operation of the vehicle in an environment as controlled by the controller. The evaluation computing system may modify a portion of the data representative of a fault associated with the controller. The evaluation computing system may run a simulation utilizing modified log data to determine whether the controller detects and/or mitigates the fault within a threshold time. Based on a determination that the controller does not detect and/or mitigate the fault within the threshold time, the evaluation computing system may determine an error associated with the controller. Based on a determination that the controller detects and/or mitigates the fault within the threshold time, the evaluation computing system may validate the controller.

Fault detection in vehicle control systems

Techniques for training a model for detecting objects in an environment are discussed herein. For example, techniques can include determining losses associated with spatial features of candidate bounding boxes output by a machine-learned (ML) model and utilizing the losses to train the ML model. Techniques may include determining candidate bounding box(es) associated with an object detected in an environment using the ML model and receiving a ground truth bounding box associated with the detected object. A yaw error loss may be determined by comparing yaw features of the candidate bounding box to the ground truth bounding box. The candidate bounding box may be axis aligned with respect to the ground truth bounding box and an intersection over union (IoU) loss may be determined based on an IoU between the axis aligned candidate bounding box and the ground truth bounding box. The ML model may be trained based on the losses.

Geometric confidence for tracking objects

A computer implemented method. Includes: obtaining, from a sensor system onboard a vehicle, a plurality of hypotheses for an orientation of an object in a vicinity of the vehicle, relative to a coordinate system of the vehicle; estimating, based on the plurality of hypotheses, values of a probability distribution for the orientation of the object at a plurality of candidate orientations; and estimating, based at least in part on the estimated values of the probability distribution at the plurality of candidate orientations, the orientation of the object relative to the coordinate system.

Object orientation estimator

Techniques to use a trained model to determine a yaw of an object are described. For example, a system may implement various techniques to generate multiple representations for an object in an environment. Each representation vary based on the technique and data used. An estimation component may estimate a representation from the multiple representations. The model may be implemented to output a yaw for the object using the multiple representations, the estimated representation, and/or additional information. The output yaw may be used to track an object, generate a trajectory, or otherwise control a vehicle.

Determining yaw with learned motion model

Predicting articulated object states

The techniques discussed herein include modifying a Kalman filter to additionally include a loss component that dampens the effect measurements with large errors (or measurements indicating states that are rather different than the predicted state) have on the Kalman filter and, in particular, the updated uncertainty and/or updated prediction. In some examples, the techniques include scaling a Kalman gain based at least in part on a loss function that is based on the innovation determined by the Kalman filter. The techniques additionally or alternatively include a reformulation of a Kalman filter that ensures that the uncertainties determined by the Kalman filter remain symmetric and positive definite.

Robust numerically stable Kalman filter for autonomous vehicles

A computer-implemented method. Includes obtaining pointwise data indicating, for a plurality of time steps, a pointwise measurement of a state of an object detected by an object detection system. Includes obtaining, from a runtime model, runtime data indicating, for the plurality of time steps, a runtime estimate of the state of the object. Includes processing, by a benchmark model, the pointwise data to determine, for the plurality of time steps, a benchmark estimate of the state of the object. Includes evaluating a metric measuring, for the plurality of time steps, a deviation between the runtime estimate and the benchmark estimate of the state of the object. Includes updating, based on the on the evaluation of the metric, the runtime model.

Growing community of inventors

Sunnyvale, CA, United States of America

Francesco Papi