Zoox, Inc.

Marin Kobilarov

Timothy Caldwell

Gowtham Garimella

Kai Zhenyu Wang

Gary Linscott

Andres Guillermo Morales Morales

Dan Xie

Joseph Funke

Ashutosh Gajanan Rege

Jefferson Bradfield Packer

Mark Jonathon McClelland

Rasmus Fonseca

Linjun Zhang

Jacob Lee Askeland

Abishek Krishna Akella

Robert Edward Somers

Joona Markus Petteri Kiiski

Jesse Sol Levinson

Ravi Verma Gogna

Ethan Miller Pronovost

Janek Hudecek

Paul Orecchio

Amanda Lee Kelly Lockwood

Vasumathi Raman

Chonhyon Park

Christopher Paxton

Jack Riley

William Anthony Silva

Rick Zhang

Lichao Ma

Matthew Van Heukelom

Xiaosi Zeng

Chuang Wang

Marc Wimmershoff

Zhenqi Huang

Adrian Michael Costantino

Yunpeng Pan

Liam Gallagher

Matthew Sheckells

Andreas Pasternak

Steven Cheng Qian

Peter Scott Schleede

Kazuhide Okamoto

David Evan Zlotnik

Jacob Patrick Thalman

Olivier Toupet

Sriram Narayanan

Jeremy Schwartz

Arian Houshmand

Dhanushka Nirmevan Kularatne

Xianan Huang

Ruikun Yu

Vincent Andreas Laurense

Ke Sun

Lingfeng Zhang

Cheng Peng

Yiteng Zhang

Xiangyu Xie

Aaron Huang

Yan Chang

A relevance filter may determine to use additional computing resources to generate predicted trajectory(ies) for a subset object(s) detected by a vehicle that depend on a candidate action for controlling the vehicle. The relevance filter may determine the subset by a memory storage and parallel computing technique that includes determining a set of vehicle states associated with a path for controlling the vehicle; determining a set of object states associated with a predicted trajectory of an object; determining a set of interaction scores based on the set of vehicle states and the set of object states; determining, based on the set of interaction scores, a trajectory importance score associated with the predicted trajectory; determining, based on the trajectory importance score, an object importance score associated with the object; and controlling the vehicle based at least in part on the object importance score and/or a new predicted trajectory for the object.

Parallel processing filter for detecting object relevance to vehicle operation planning

Systems and techniques for whether to associate predicted trajectories of objects detected in an environment with candidate vehicle trajectories are described. A vehicle traversing an environment may have several candidate actions that may be used to control the vehicle. Predicted object trajectories may be excluded from further processing based on a relevancy score determination. Various factors may be evaluated to determine the relevancy score associated with a predicted object trajectory and candidate vehicle action pair that indicated the impact of the predicted object trajectory and candidate vehicle action. A resultant trajectory may be determined using only those relevant predicted object trajectories.

Object relevance filtering

Techniques for storing predicted state using a linear array and for querying the linear array to evaluate a proposed trajectory are described herein. In some cases, a first processing unit (e.g., a central processing unit (CPU)) determines prediction data associated with an object in the vehicle environment and sends the prediction data to a second processing unit (e.g., a graphics processing unit (GPU)). In some cases, after the second processing unit receives the prediction data, the second processing unit stores the prediction data using a linear array. In some cases, the second processing unit can use the linear array to evaluate a proposed trajectory for the vehicle and determine whether to validate the proposed trajectory for the vehicle. In some cases, based on the validation of the proposed trajectory by the second processing unit, the vehicle is controlled based on the proposed trajectory.

Evaluating a vehicle trajectory using a linear array query

Techniques for accurately predicting and avoiding collisions with objects detected in an environment of a vehicle are discussed herein. A vehicle computing device can implement a model to output data indicating costs for potential intersection points between the object and the vehicle in the future. The model may employ a control policy and a time-step integrator to determine whether an object may intersect with the vehicle, in which case the techniques may include predicting vehicle actions by the vehicle computing device to control the vehicle.

Active prediction based on object trajectories

Techniques are discussed herein for determining optimal driving trajectories for autonomous vehicles in complex multi-agent driving environments. A baseline trajectory may be perturbed and parameterized into a vector of vehicle states associated with different segments (or portions) of the trajectory. Such a vector may be modified to ensure the resultant perturbed trajectory is kino-dynamically feasible. The vectorized perturbed trajectory may be input, including a representation of the current driving environment and additional agents, into a prediction model trained to output a predicted future driving scene. The predicted future driving scene, including predicted future states for the vehicle and predicted trajectories for the additional agents in the environment, may be evaluated to determine costs associated with each perturbed trajectory. Based on the determined costs, the optimization algorithm may determine subsequent perturbations and/or the optimal trajectory for controlling the vehicle in the driving environment.

Trajectory optimization in multi-agent environments

A machine-learned architecture for estimating the cost determined by a cost function for a prediction node of a tree search for exploring potential paths for controlling a vehicle may include two portions: a set up portion that includes models trained to process static data and a second portion that processes dynamic object data. The respective portions of the architecture may comprise various models that determine intermediate outputs that may be projected into a space associated with estimated cost. That estimated cost may identify an estimate of an output of the cost function for paths that are based on a particular prediction node of the tree search.

Machine-learned cost estimation in tree search trajectory generation for vehicle control

Techniques for determining relevant objects in an environment around a vehicle are discussed herein. Sensor data can be captured and input to a model to determine predicted object trajectories and associated probabilities. For a candidate trajectory for the vehicle to follow, regions of overlap between the vehicle trajectory and the predicted object trajectories can be determined. An interaction graph can be generated representing interactions between the candidate trajectory and the predicted object trajectories. The interaction graph can be sampled based on the probabilities and/or other characteristics. Samples from the interaction graph can be used to determine relevant objects in a driving scenario for a vehicle traversing an environment. Based on the sample, the vehicle may generate a relevancy score associated with the sample. The vehicle may determine that a number of the most relevant sample scenarios may be sent to a prediction component and control of the vehicle.

Identifying relevant objects within an environment

Techniques are discussed for predicting an occupancy of visible region of an environment. For instance, a vehicle may generate sensor data representing an environment. The vehicle may then analyze the sensor data to determine an occluded region of the environment a visible region of the environment. Additionally, the vehicle may determine at least one prediction probability associated with occupancy of the visible region over a future period of time. In some instances, the vehicle determines the at least one prediction probability by inputting data representing at least the occluded region and the visible region into a machine learned model and receiving the at least one prediction probability from the machine learned model. Using the at least one prediction probability, the vehicle may then determine and perform one or more actions.

Predicting occupancy of objects in occluded regions

Techniques relating to determining regions based on intents of objects are described. In an example, a computing device onboard a first vehicle can receive sensor data associated with an environment of the first vehicle. The computing device can determine, based on the sensor data, a region associated with a second vehicle proximate the first vehicle that is to be occupied by the second vehicle while the vehicle performs a maneuver. Further, the computing device can determine an instruction for controlling the first vehicle based at least in part on the region.

Spatial prediction

Systems and techniques for determining a sideslip vector for a vehicle that may have a direction that is different from that of a heading vector for the vehicle. The sideslip vector in a current vehicle state and sideslip vectors in predicted vehicles states may be used to determine paths for a vehicle through an environment and trajectories for controlling the vehicle through the environment. The sideslip vector may be based on a vehicle position that is the center point of the wheelbase of the vehicle and may include lateral velocity, facilitating the control of four-wheel steered vehicle while maintaining the ability to control two-wheel steered vehicles.

Determining vehicle position using sideslip vector

Techniques for representing sensor data and predicted behavior of various objects in an environment are described herein. For example, an autonomous vehicle can represent prediction probabilities as an uncertainty model that may be used to detect potential collisions, define a safe operational zone or drivable area, and to make operational decisions in a computationally efficient manner. The uncertainty model may represent a probability that regions within the environment are occupied using a heat map type approach in which various intensities of the heat map represent a likelihood of a corresponding physical region being occupied at a given point in time.

Object uncertainty models to assist with drivable area determinations

Command determination for controlling a vehicle, such as an autonomous vehicle, is described. In an example, individual requests for controlling the vehicle relative to each of multiple objects or conditions in an environment are received (substantially simultaneously) and based on the request type and/or additional information associated with a request, command controllers can determine control commands (e.g., different accelerations, steering angles, steering rates, and the like) associated with each of the one or more requests. The command controllers may have different controller gains (which may be based on functions of distance, distance ratios, time to estimated collisions, etc.) for determining the controls and a control command may be determined based on the all such determined controls.

Vehicle control

Generating a lane reference from a roadway shape and/or generating a trajectory for controlling an autonomous vehicle may include determining a predicted state of the lane reference and/or a candidate trajectory by an integrator. The disclosed integrator is implemented as a numerical integrator in predominantly closed-form that is able to avoid singularities while maintaining no approximation error. The disclosed integrator is also more robust to poor initial estimations, high curvature roadways, and zero-velocity conditions.

Route-relative trajectory numerical integrator and controller using the same

An imitation learning-based machine-learned (ML) model to augment or replace the prediction and/or planner components of an autonomous vehicle may be trained using a two stage and multi-discipline approach. A first stage of training may include training the ML component to output a predicted action associated with a target vehicle and modifying the ML component to reduce a difference between the predicted action and the observed action taken by the target vehicle. A second stage may use reinforcement learning to further tun the ML component. The resultant model may be used on its own, with enough training data, or to rank or weight candidate trajectories generated by a planning component of the vehicle. The ML component may provide embeddings of environment features to first transformer(s) that output to a long short-term memory that outputs to second transformer(s) to determine the predicted action.

Machine-learned component hybrid training and assistance of vehicle trajectory generation

Techniques for determining unified futures of objects in an environment are discussed herein. Techniques may include determining a first feature associated with an object in an environment and a second feature associated with the environment and based on a position of the object in the environment, updating a graph neural network (GNN) to encode the first feature and second feature into a graph node representing the object and encode relative positions of additional objects in the environment into one or more edges attached to the node. The GNN may be decoded to determine a distribution of predicted positions for the object in the future that meet a criterion, allowing for more efficient sampling. A predicted position of the object in the future may be determined by sampling from the distribution.

Focusing prediction distribution output for efficient sampling

Growing community of inventors

Mountain View, CA, United States of America

Marin Kobilarov