Zoox, Inc.

Kai Zhenyu Wang

Gowtham Garimella

James William Vaisey Philbin

Marin Kobilarov

Andres Guillermo Morales Morales

Sarah Tariq

Samir Parikh

Vasiliy Karasev

Ethan Miller Pronovost

Andrew Scott Crego

Tencia Lee

Michael Haggblade

Xiaosi Zeng

Chuang Wang

Feng Tian

Yi Xu

Cooper Stokes Sloan

Benjamin John Sapp

Gerrit Bagschik

Andraz Kavalar

Benjamin Isaac Mattinson

Xi Joey Hong

Tianyi Cai

Pengfei Duan

Linjun Zhang

Abishek Krishna Akella

Bertrand Robert Douillard

Mahsa Ghafarianzadeh

Rick Zhang

Juhana Kangaspunta

Matthew Van Heukelom

Rasmus Fonseca

Christopher James Gibson

Kenneth Michael Siebert

Muhammad Farooq Rama

Chunlei Dai

Greg Woelki

Jonathan Philip Wai Wah Chan

Yousef Ali Emam

Xiaobo Ren

Woodrow Zhouyuan Wang

A machine-learned architecture for determining whether an object is relevant to a vehicle's action planning may comprise a convolutional neural network, graph neural network, and/or multi-layer perceptron that may determine a relevance score associated with an object that indicates indicating whether an object is likely to impact operation(s) of a vehicle. In some examples, the machine-learned architecture may use scene information and/or an object track to determine the relevance score.

Machine-learned model for detecting object relevance to vehicle operation planning

Techniques for determining a response of a simulated vehicle to a simulated object in a simulation are discussed herein. Log data captured by a physical vehicle in an environment can be received. Object data representing an object in the log data can be used to instantiate a simulated object in a simulation to determine a response of a simulated vehicle to the simulated object. Additionally, one or more trajectory segments in a trajectory library representing the log data can be determined and instantiated as a trajectory of the simulated object in order to increase the accuracy and realism of the simulation.

Hybrid log simulated driving

System for simulating an autonomous vehicle using a trajectory library and log data

Techniques are discussed for predicting an occupancy of visible region of an environment. For instance, a vehicle may generate sensor data representing an environment. The vehicle may then analyze the sensor data to determine an occluded region of the environment a visible region of the environment. Additionally, the vehicle may determine at least one prediction probability associated with occupancy of the visible region over a future period of time. In some instances, the vehicle determines the at least one prediction probability by inputting data representing at least the occluded region and the visible region into a machine learned model and receiving the at least one prediction probability from the machine learned model. Using the at least one prediction probability, the vehicle may then determine and perform one or more actions.

Predicting occupancy of objects in occluded regions

Techniques relating to determining regions based on intents of objects are described. In an example, a computing device onboard a first vehicle can receive sensor data associated with an environment of the first vehicle. The computing device can determine, based on the sensor data, a region associated with a second vehicle proximate the first vehicle that is to be occupied by the second vehicle while the vehicle performs a maneuver. Further, the computing device can determine an instruction for controlling the first vehicle based at least in part on the region.

Spatial prediction

Techniques are discussed for determining prediction probabilities of an object based on a top-down representation of an environment. Data representing objects in an environment can be captured. Aspects of the environment can be represented as map data. A multi-channel image representing a top-down view of object(s) in the environment can be generated based on the data representing the objects and map data. The multi-channel image can be used to train a machine learned model by minimizing an error between predictions from the machine learned model and a captured trajectory associated with the object. Once trained, the machine learned model can be used to generate prediction probabilities of objects in an environment, and the vehicle can be controlled based on such prediction probabilities.

Trajectory prediction on top-down scenes and associated model

Techniques relating to monitoring map consistency are described. In an example, a monitoring component associated with a vehicle can receive sensor data associated with an environment in which the vehicle is positioned. The monitoring component can generate, based at least in part on the sensor data, an estimated map of the environment, wherein the estimated map is encoded with policy information for driving within the environment. The monitoring component can then compare first information associated with a stored map of the environment with second information associated with the estimated map to determine whether the estimated map and the stored map are consistent. Component(s) associated with the vehicle can then control the object based at least in part on results of the comparing.

Map consistency checker

Techniques for determining unified futures of objects in an environment are discussed herein. Techniques may include determining a first feature associated with an object in an environment and a second feature associated with the environment and based on a position of the object in the environment, updating a graph neural network (GNN) to encode the first feature and second feature into a graph node representing the object and encode relative positions of additional objects in the environment into one or more edges attached to the node. The GNN may be decoded to determine a distribution of predicted positions for the object in the future that meet a criterion, allowing for more efficient sampling. A predicted position of the object in the future may be determined by sampling from the distribution.

Focusing prediction distribution output for efficient sampling

Techniques for determining unified futures of objects in an environment are discussed herein. Techniques may include determining a first feature associated with an object in an environment and a second feature associated with the environment and based on a position of the object in the environment, updating a graph neural network (GNN) to encode the first feature and second feature into a graph node representing the object and encode relative positions of additional objects in the environment into one or more edges attached to the node. The GNN may be decoded to determine a distribution of predicted positions for the object in the future. A predicted position of the object at a subsequent timestep may be determined by sampling from the distribution of predicted positions according to various sampling strategies. Alternatively, the predicted position of the object may be overwritten using a candidate position of the object.

Prediction sampling techniques

Techniques for determining unified futures of objects in an environment are discussed herein. Techniques may include determining a first feature associated with an object in an environment and a second feature associated with the environment and based on a position of the object in the environment, updating a graph neural network (GNN) to encode the first feature and second feature into a graph node representing the object and encode relative positions of additional objects in the environment into one or more edges attached to the node. The GNN may be decoded to determine a predicted position of the object at a subsequent timestep. Further, a predicted trajectory of the object may be determined using predicted positions of the object at various timesteps.

Encoding relative object information into node edge features

Techniques are disclosed for component verification for complex systems. The techniques may include receiving log data, obtaining ground truth data based on the log data and determining an outcome at least in part by simulating a prediction by a prediction component based on the log data and the ground truth data. The techniques may further include simulating a second prediction by the prediction component based on the ground truth data, determining whether the second prediction resulted in the negative outcome of the scenario and determining the disengagement event is attributable to a perception component of the autonomous operation system at least partly in response to determining the second prediction based on the ground truth data did not result in the negative outcome.

Component verification system

Techniques for determining a prediction probability associated with a disengagement event are discussed herein. A first prediction probability can include a probability that a safety driver associated with a vehicle (such as an autonomous vehicle) may assume control over the vehicle. A second prediction probability can include a probability that an object in an environment is associated the disengagement event. Sensor data can be captured and represented as a top-down representation of the environment. The top-down representation can be input to a machine learned model trained to output prediction probabilities associated with a disengagement event. The vehicle can be controlled based the prediction probability and/or the interacting object probability.

Disengagement prediction for vehicles

Techniques for top-down scene discrimination are discussed. A system receives scene data associated with an environment proximate a vehicle. The scene data is input to a convolutional neural network (CNN) discriminator trained using a generator and a classification of the output of the CNN discriminator. The CNN discriminator generates an indication of whether the scene data is a generated scene or a captured scene. If the scene data is data generated scene, the system generates a caution notification indicating that a current environmental situation is different from any previous situations. Additionally, the caution notification is communicated to at least one of a vehicle system or a remote vehicle monitoring system.

Top-down scene discrimination

Techniques for modeling the probability distribution of errors in perception systems are discussed herein. For example, techniques may include modeling error distribution for attributes such as position, size, pose, and velocity of objects detected in an environment, and training a mixture model to output specific error probability distributions based on input features such as object classification, distance to the object, and occlusion. The output of the trained model may be used to control the operation of a vehicle in an environment, generate simulations, perform collision probability analyses, and to mine log data to detect collision risks.

Perception error modeling

Techniques for top-down scene generation are discussed. A generator component may receive multi-dimensional input data associated with an environment. The generator component may generate, based at least in part on the multi-dimensional input data, a generated top-down scene. A discriminator component receives the generated top-down scene and a real top-down scene. The discriminator component generates binary classification data indicating whether an individual scene in the scene data is classified as generated or classified as real. The binary classification data is provided as a loss to the generator component and the discriminator component.

Top-down scene generation

Growing community of inventors

Foster City, CA, United States of America

Kai Zhenyu Wang