Zoox, Inc.

Jefferson Bradfield Packer

William Anthony Silva

Marin Kobilarov

Timothy Caldwell

Collin MacGregor

Rick Zhang

Sy Kelly Olson

Gowtham Garimella

Abishek Krishna Akella

Marc Wimmershoff

Dan Xie

Zhenqi Huang

Robert Edward Somers

Andrew Lewis King

Andreas Pasternak

Sarah Tariq

Joseph Funke

Gary Linscott

Janek Hudecek

Andreas Christian Reschka

Lichao Ma

Matthew Van Heukelom

Rasmus Fonseca

David Martins Belo

Ruikun Yu

Yiteng Zhang

Aaron Huang

A vehicle computing system may implement techniques to control a vehicle to avoid collisions between the vehicle and agents (e.g., dynamic objects) in an environment. The techniques may include generating a representation of a path of the vehicle through an environment as a polygon. The vehicle computing system may compare the two-dimensional path with a trajectory of an agent determined using sensor data to determine a collision zone between the vehicle and the agent. The vehicle computing system may determine a risk of collision based on predicted velocities and probable accelerations of the vehicle and the agent approaching and traveling through the collision zone. Based at least in part on the risk of collision, the vehicle computing system may cause the vehicle to perform an action.

Collision prediction and avoidance for vehicles

Techniques are discussed for interaction probabilities associated with regions of an environment around a vehicle. An interaction probability of a region may indicate a likelihood an object positioned at the region will interact with the vehicle. A top-down multi-channel image representing a top-down view of the environment and objects therein may be generated and input to a machine learned (ML) model. The ML model may output a probability map, a portion of the probability map comprising a region and an interaction probability associated with the region that indicates a likelihood objects positioned at the region will interact with the vehicle. A priority for resource assignment or analysis may be determined based on the interaction probability for an object positioned in the region. Control of the vehicle may be performed based at least in part on the priority for resource assignment or analysis.

Machine learned interaction prediction from top-down representation

Automating reinforcement learning for autonomous vehicles may include assigning a probability with a scenario and varying that probability based at least in part on changes in performance by the autonomous vehicle associated with that scenario. The amount of time and computational bandwidth required to train a machine-learned component of an autonomous vehicle and the accuracy of the machine-learned component may be improved by determining a reward for performance of the autonomous vehicle in a scenario based at least in part on an severity metric. The impact severity metric may be determined based at least in part on a velocity, angle, and/or interaction area associated with the impact.

Automated reinforcement learning scenario variation and impact penalties

A reward determined as part of a machine learning technique, such as reinforcement learning, may be used to control an adversarial agent in a simulation such that a component for controlling motion of the adversarial agent is trained to reduce the reward. Training the adversarial agent component may be subject to one or more constraints and/or may be balanced against one or more additional goals. Additionally or alternatively, the reward may be used to alter scenario data so that the scenario data reduces the reward, allowing the discovery of difficult scenarios and/or prospective events.

Adversarial agent controls generation and problematic scenario forecasting

Drive envelope determination is described. In an example, a vehicle can capture sensor data while traversing an environment and can provide the sensor data to computing system(s). The sensor data can indicate agent(s) in the environment and the computing system(s) can determine, based on the sensor data, a planned path through the environment relative to the agent(s). The computing system(s) can also determine lateral distance(s) to the agent(s) from the planned path. In an example, the computing system(s) can determine modified distance(s) based at least in part on the lateral distance(s) and information about the agents. The computing system can determine a drive envelope based on the modified distance(s) and can determine a trajectory in the drive envelope.

Drive envelope determination

A vehicle computing system may implement techniques to determine relevance of objects detected in an environment to a vehicle operating in the environment (e.g., whether an object could potentially impact the vehicle's ability to safely travel). The techniques may include determining an initial location and trajectory associated with a detected object and determining whether the detected object may intersect a path polygon (e.g., planned path with a safety buffer) of the vehicle. A path intersection may include the detected object intersecting the path polygon or sharing a road segment with the vehicle at a substantially similar height as the vehicle (e.g., on a similar plane). Based on a determination that the detected object does not intersect the planned path of the vehicle, the vehicle computing system may determine that the detected object is irrelevant to the vehicle and may disregard the detected object in vehicle control planning considerations.

Object relevance determination

A vehicle may include a primary system and a secondary system to validate operation of the primary system and to control the vehicle to avoid collisions. For example, the secondary system may receive multiple trajectories from the primary system, such as a primary trajectory and a secondary, contingent, trajectory associated with a deceleration or other maneuver. The secondary system may determine if a trajectory is associated with a potential collision, if the trajectory is consistent with a current or previous pose, if the trajectory is compatible with a capability of the vehicle, etc. The secondary system may select the primary trajectory if valid, the secondary trajectory if the primary trajectory is invalid, or another trajectory generated by the secondary system if the primary trajectory and the secondary trajectory are invalid. If no valid trajectory is determined, the vehicle may decelerate at a maximum rate.

Collision avoidance system with trajectory validation

A vehicle computing system may implement techniques to predict behavior of objects detected by a vehicle operating in the environment. The techniques may include determining a feature with respect to a detected objects (e.g., likelihood that the detected object will impact operation of the vehicle) and/or a location of the vehicle and determining based on the feature a model to use to predict behavior (e.g., estimated states) of proximate objects (e.g., the detected object). The model may be configured to use one or more algorithms, classifiers, and/or computational resources to predict the behavior. Different models may be used to predict behavior of different objects and/or regions in the environment. Each model may receive sensor data as an input, and output predicted behavior for the detected object. Based on the predicted behavior of the object, a vehicle computing system may control operation of the vehicle.

Hyper planning based on object and/or region

Techniques for generating simulations for evaluating a performance of a controller of an autonomous vehicle are described. A computing system may evaluate the performance of the controller to navigate the simulation and respond to actions of one or more objects (e.g., other vehicles, bicyclists, pedestrians, etc.) in a simulation. Actions of the objects in the simulation may be controlled by the computing system (e.g., by an artificial intelligence) and/or one or more users inputting object controls, such as via a user interface. The computing system may calculate performance metrics associated with the actions performed by the vehicle in the simulation as directed by the autonomous controller. The computing system may utilize the performance metrics to verify parameters of the autonomous controller (e.g., validate the autonomous controller) and/or to train the autonomous controller utilizing machine learning techniques to bias toward preferred actions.

Vehicle controller simulations

Systems and methods are described for performing one or more validity checks on potential trajectories for a device, such as an autonomous vehicle, to navigate. In some examples, a potential trajectory may be validated based on whether it is consistent with a current trajectory the vehicle is navigating such that the potential and current trajectories are not too different, whether the vehicle can feasibly or kinematically navigate to the potential trajectory from a current state, whether the potential trajectory was punctual or received within a time period of a prior trajectory, and/or whether the potential trajectory passes a staleness check, such that it was created within a certain time period. In some examples, determining whether a potential trajectory is feasibly may include updating a set of feasibility limits based on one or more operational characteristics of statuses of subsystems of the vehicle.

Feasibility validation for vehicle trajectory selection

The present disclosure is directed to performing one or more validity checks on potential trajectories for a device, such as an autonomous vehicle, to navigate. In some examples, a potential trajectory may be validated based on whether it is consistent with a current trajectory the vehicle is navigating such that the potential and current trajectories are not too different, whether the vehicle can feasibly or kinematically navigate to the potential trajectory from a current state, whether the potential trajectory was punctual or received within a time period of a prior trajectory, and/or whether the potential trajectory passes a staleness check, such that it was created within a certain time period. In some examples, determining whether a potential trajectory is feasibly may include updating a set of feasibility limits based on one or more operational characteristics of statuses of subsystems of the vehicle.

System and method for trajectory validation

Consistency validation for vehicle trajectory selection

A method and apparatus for selecting among a plurality of trajectories comprises receiving at least a first and second trajectory, each having an associated level in a trajectory hierarchy, determining whether the first trajectory is viable, determining a present level limit for trajectory selection, and if the first trajectory is viable and its level does not exceed the present level limit, executing the first trajectory. If the first trajectory is not viable or the associated level of the first trajectory exceeds the present level limit, the second trajectory is executed if it is viable and its associated level does not exceed the present level limit. A state variable is set such that when a trajectory lower in a trajectory hierarchy is executed, a trajectory selector waits for a message from a monitor or an operator before returning to the use of higher level trajectories.

System and method for effecting a safety stop release in an autonomous vehicle

Techniques are discussed for evaluating trajectories based on risk associated with the trajectories with respect to predicted locations of objects in an environment. A vehicle can capture sensor data of an environment, which may include object(s) separate from the vehicle, such as another vehicle or a pedestrian. A prediction system can output a discretized probability distribution comprising prediction probabilities associated with possible locations of the object in the future. Heat maps, as an example discretized probability distribution, can represent one or more objects. Trajectories can be generated for the vehicle to follow in the environment. An overlap between a region of the vehicle along a trajectory and the heat map can be determined, and a probability associated with the overlap can represent a risk associated with a trajectory navigating through the environment. The vehicle can be controlled based on risks associated with the various trajectories.

Growing community of inventors

San Francisco, CA, United States of America

Jefferson Bradfield Packer