Zoox, Inc.

Disney Enterprises, Inc.

Timothy Caldwell

Marin Kobilarov

Ravi Verma Gogna

Dan Xie

Rick Zhang

William Anthony Silva

Ashutosh Gajanan Rege

Gary Linscott

Jacob Lee Askeland

Jefferson Bradfield Packer

Robert Edward Somers

Joona Markus Petteri Kiiski

Jesse Sol Levinson

Abishek Krishna Akella

Paul Orecchio

Amanda Lee Kelly Lockwood

Vasumathi Raman

Christopher Paxton

Meredith James Goldman

Frank Reinaldo Ramirez

Matthew Van Heukelom

Matthew Miller Young

Joseph Funke

Rasmus Fonseca

Xianan Huang

Joseph Lorenzetti

Matthew Sheckells

Lanny Starkes Smoot

Gary W Schnuckle

Linjun Zhang

Gowtham Garimella

Lichao Ma

Chonhyon Park

Yunpeng Pan

Arian Houshmand

Seyed Mahdi Shamsi

Ke Sun

Clark Michael Davenport

Shahriar Sefati

Anjali Lakshmi Pemmaraju

Cheng Peng

Techniques for determining adversarial cost(s) representing an object in an environment are described herein. For example, a vehicle may generate candidate action(s) (or trajectories) and input such candidate actions into a tree structure. The vehicle can determine a control trajectory for the vehicle to follow based a determining which of the candidate actions to follow at each layer in the tree structure. The vehicle can determine which candidate action to follow based on determining adversarial cost(s). To generate the adversarial cost(s), the vehicle can detect an object and determine associated object trajectories. Based on a likelihood that the object will deviate from a predicted object trajectory, the vehicle can determine one or more sub-factors that may be combined to generate the adversarial cost(s). The vehicle can be controlled based on the adversarial cost(s).

Determining adversarial cost based on generated deviations

A vehicle computing system may identify an obstruction along a route of travel and may connect to a service computing device for guidance. The service computing device may include a guidance system configured to receive waypoint and/or orientation input from an operator. The operator may evaluate the scenario and determine one or more waypoints and/or associated orientations for the vehicle to navigate the scenario. In some examples, the guidance system may validate the waypoint(s) and/or associated orientation(s). The service computing device may send the waypoint(s) and/or associated orientation(s) to the vehicle computing system. The vehicle computing system may validate the waypoint(s) and/or associated orientation(s) and, based on the validation, control the vehicle according to the input. Based on a determination that the vehicle has navigated the scenario, the guidance system may release vehicle guidance back to the vehicle computing system.

Remote vehicle guidance

Techniques for determining a vehicle trajectory that causes a vehicle to navigate in an environment relative to one or more objects are described herein. For example, the techniques may include a computing device determining a decision tree having nodes to represent different object intents and/or nodes to represent vehicle actions at a future time. A tree search algorithm can search the decision tree to evaluate potential interactions between the vehicle and the one or more objects over a time period, and output a vehicle trajectory for the vehicle. The vehicle trajectory can be sent to a vehicle computing device for consideration during vehicle planning, which may include simulation.

Trajectory prediction based on a decision tree

Trajectory generation and/or execution architecture is described. In an example, a first signal can be determined at a first frequency, wherein the first signal comprises information associated with causing the system to move to a location. Further, a second signal can be determined at a second frequency different from the first frequency and based at least in part on the first signal. A system can be controlled to move to the location, based at least in part on the second signal.

Trajectory generation and execution architecture

Trajectory generation for controlling motion or other behavior of an autonomous vehicle may include alternately determining a candidate action and predicting a future state based on that candidate action. The technique may include determining a cost associated with the candidate action that may include an estimation of a transition cost from a current or former state to a next state of the vehicle. This cost estimate may be a lower bound cost or an upper bound cost and the tree search may alternately apply the lower bound cost or upper bound cost exclusively or according to a ratio or changing ratio. The prediction of the future state may be based at least in part on a machine-learned model's classification of a dynamic object as being a reactive object or a passive object, which may change how the dynamic object is modeled for the prediction.

Vehicle trajectory control using a tree search

Drive envelope determination is described. In an example, a vehicle can capture sensor data while traversing an environment and can provide the sensor data to computing system(s). The sensor data can indicate agent(s) in the environment and the computing system(s) can determine, based on the sensor data, a planned path through the environment relative to the agent(s). The computing system(s) can also determine lateral distance(s) to the agent(s) from the planned path. In an example, the computing system(s) can determine modified distance(s) based at least in part on the lateral distance(s) and information about the agents. The computing system can determine a drive envelope based on the modified distance(s) and can determine a trajectory in the drive envelope.

Drive envelope determination

A method for autonomously operating a driverless vehicle along a path between a first geographic location and a destination may include receiving communication signals from the driverless vehicle. The communication signals may include sensor data from the driverless vehicle and data indicating occurrence of an event associated with the path. The communication signals may also include data indicating that a confidence level associated with the path is less than a threshold confidence level due to the event. The method may also include determining, via a teleoperations system, a level of guidance to provide the driverless vehicle based on data associated with the communication signals, and transmitting teleoperations signals to the driverless vehicle. The teleoperations signals may include guidance to operate the driverless vehicle according to the determined level of guidance, so that a vehicle controller maneuvers the driverless vehicle to avoid, travel around, or pass through the event.

Interactions between vehicle and teleoperations system

Techniques for generating simulations for evaluating a performance of a controller of an autonomous vehicle are described. A computing system may evaluate the performance of the controller to navigate the simulation and respond to actions of one or more objects (e.g., other vehicles, bicyclists, pedestrians, etc.) in a simulation. Actions of the objects in the simulation may be controlled by the computing system (e.g., by an artificial intelligence) and/or one or more users inputting object controls, such as via a user interface. The computing system may calculate performance metrics associated with the actions performed by the vehicle in the simulation as directed by the autonomous controller. The computing system may utilize the performance metrics to verify parameters of the autonomous controller (e.g., validate the autonomous controller) and/or to train the autonomous controller utilizing machine learning techniques to bias toward preferred actions.

Vehicle controller simulations

Techniques described herein are directed to classifying lanes in an environment of a vehicle for, for example, performing lane handling. In an example, system(s) of a vehicle can determine a signal indicative of a presence of the vehicle in a lane of a drivable surface in an environment within which the vehicle is located. The system(s) can determine, based at least in part on the signal, a classification of the lane as at least one of occupied (an object is at least partially in the lane), unoccupied (no object in the lane), and/or established (e.g., where an object has established a priority in the lane). The system(s) can control the vehicle based at least in part on the classification of the lane to improve safety in scenarios, for example, including merging.

Lane classification for improved vehicle handling

A vehicle computing system may identify a scenario in an environment that violates an operating constraint. The vehicle computing system may request remote guidance from a guidance system of a service computing device. The vehicle computing system may receive input from the guidance system including one or more waypoints and/or associated orientations for the vehicle to navigate through the scenario. The vehicle computing system may be configured to validate the input. A validation may include processing the input to determine whether the waypoint(s) and/or orientation(s) associated therewith may cause the vehicle to violate a safety protocol. Based on a determination that the input will not cause the vehicle to violate the safety protocol, the vehicle computing system may control the vehicle according to the input, such as by causing a drive system to operate the vehicle to each waypoint at the associated orientation.

Vehicle control and guidance

Techniques for determining a trajectory for an autonomous vehicle are described herein. In general, determining a route can include utilizing a search algorithm such as Monte Carlo Tree Search (MCTS) to search for possible trajectories, while using temporal logic formulas, such as Linear Temporal Logic (LTL), to validate or reject the possible trajectories. Trajectories can be selected based on various costs and constraints optimized for performance. Determining a trajectory can include determining a current state of the autonomous vehicle, which can include determining static and dynamic symbols in an environment. A context of an environment can be populated with the symbols, features, predicates, and LTL formula. Rabin automata can be based on the LTL formula, and the automata can be used to evaluate various candidate trajectories. Nodes of the MCTS can be generated and actions can be explored based on machine learning implemented as, for example, a deep neural network.

Trajectory generation using temporal logic and tree search

Techniques described herein relate to lane handling, for instance to enable vehicles to perform turns without colliding into oncoming vehicles and/or bicycles in other lanes. System(s) associated with a vehicle can access sensor data and/or map data associated with an environment within which the vehicle is positioned in a first lane. The system(s) can determine that the vehicle is to perform a turn and a start of a second lane associated with the turn or a merging zone associated with the second lane. The system(s) can determine a location of the vehicle relative to the start of the second lane or the merging zone associated with the second lane and, based partly on the location, can cause the vehicle to merge into the second lane prior to performing the turn.

Lane handling for merge prior to turn

A vehicle computing system may implement techniques to determine an action for a vehicle to take based on a cost associated therewith. The cost may be based in part on the effect of the action on an object (e.g., another vehicle, bicyclist, pedestrian, etc.) operating in the environment. The vehicle computing system may detect the object based on sensor data and determine an object trajectory based on a predicted reaction of the object to the vehicle performing the action. The vehicle computing system may determine costs associated with safety, comfort, progress, and/or operating rules for each action the vehicle could take based on the action and/or the predicted object trajectory. In some examples, the lowest cost action may be selected for the vehicle to perform.

Cost-based path determination

Growing community of inventors

Mountain View, CA, United States of America

Timothy Caldwell