Waymo LLC

Khaled Refaat

Kai Ding

Stephane Ross

Wei Chai

Stéphane Ross

Benjamin Sapp

Haoyu Chen

Junhua Mao

Xinwei Shi

Zhishuai Zhang

Minfa Wang

Jeonhyung Kang

Dragomir Dimitrov Anguelov

Balakrishnan Varadarajan

Tian Lan

Benjamin James Caine

Alisha Saxena

Jonathan Chandler Stroud

Liting Sun

Jiachen Li

Scott Morgan Ettinger

Kratarth Goel

Chi Pang Lam

Rebecca Dawn Roelofs

Aurick Qikun Zhou

Nigamaa Nayakanti

Ahmed Said Mohammed Hefny

Rami Al-Rfou

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent behavior prediction using keypoint data. One of the methods includes obtaining data characterizing a scene in an environment, the data comprising: (i) context data comprising data characterizing historical trajectories of a plurality of agents up to the current time point; and (ii) keypoint data for a target agent; processing the context data using a context data encoder neural network to generate a context embedding for the target agent; processing the keypoint data using a keypoint encoder neural network to generate a keypoint embedding for the target agent; generating a combined embedding for the target agent from the context embedding and the keypoint embedding; and processing the combined embedding using a decoder neural network to generate a behavior prediction output for the target agent that characterizes predicted behavior of the target agent after the current time point.

Pedestrian behavior prediction with 3D human keypoints

Methods, systems, and apparatus for generating trajectory predictions for one or more target agents. In one aspect, a system comprises one or more computers configured to obtain scene context data characterizing a scene in an environment at a current time point, where the scene includes multiple agents that include a target agent and one or more context agents, and the scene context data includes respective context data for each of multiple different modalities of context data. The one or more computers then generate an encoded representation of the scene in the environment that includes one or more embeddings and process the encoded representation of the scene context data using a decoder neural network to generate a trajectory prediction output for the target agent that predicts a future trajectory of the target after the current time point.

Trajectory prediction using efficient attention neural networks

Methods, systems, and apparatus for predicting future trajectories of agents in an environment. In one aspect, a system comprises one or more computers configured to receive a data set comprising multiple training examples. The training examples include scene data comprising respective agent data for multiple agents and a ground truth trajectory for a target agent that represents ground truth motion of the target agent after a corresponding time point. The one or more computers obtain data identifying one or more of the multiple agents as non-causal agents for each training example. A non-causal agent is an agent whose states do not cause the ground truth trajectory for the target agent to change. The one or more computers generate a respective modified training example from each of the multiple training examples.

Robust behavior prediction neural networks through non-causal agent based augmentation

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for anomaly estimation for behavior predictions using a neural network. One of the methods includes receiving data characterizing a scene that includes an agent in an environment. A behavior prediction input generated from the data is processed using a behavior prediction model. The behavior prediction model is configured to process the behavior prediction input to generate a predicted probability distribution over a plurality of possible behaviors for the agent. An anomaly estimation input generated from the data is processed using an anomaly estimation model. The anomaly estimation model is configured to process the anomaly estimation input to generate a prediction error for the predicted probability distribution. The prediction error indicates an error between the predicted probability distribution generated by the behavior prediction model and another predicted probability distribution generated by another behavior prediction model.

Anomaly estimation neural networks for behavior prediction

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using context-sensitive fusion.

Agent trajectory prediction using context-sensitive fusion

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium that determine yield behavior for an autonomous vehicle, and can include identifying an agent that is in a vicinity of an autonomous vehicle navigating through a scene at a current time point. Scene features can be obtained and can include features of (i) the agent and (ii) the autonomous vehicle. An input that can include the scene features can be processed using a first machine learning model that is configured to generate (i) a crossing intent prediction that includes a crossing intent score that represents a likelihood that the agent intends to cross a roadway in a future time window after the current time, and (ii) a crossing action prediction that includes a crossing action score that represents a likelihood that the agent will cross the roadway in the future time window after the current time.

Pedestrian crossing intent yielding

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using slice-based dynamic neural networks. One of the methods includes receiving a new input for processing by a neural network that includes a first conditional neural network layer that has a set of shared parameters and a respective set of slice parameters for each of a plurality of slices. One or more slices to which the new input belongs are identified. The new input is processed to generate a network output, including: receiving a layer input to the first conditional neural network layer; and processing the layer input using the set of shared parameters, the respective one or more sets of slice parameters for the identified one or more slices, but not the respective sets of slice parameters for any other slices to which the new input does not belong.

Slice-based dynamic neural networks

Aspects of the disclosure provide for controlling a vehicle in an autonomous driving mode. For instance, sensor data for an object as well as a plurality of predicted trajectories may be received. Each predicted trajectory may represent a plurality of possible future locations for the object. A grid including a plurality of cells, each being associated with a geographic area, may be generated. Probabilities that the object will enter the geographic area associated with each of the plurality of cells over a period of time into the future may be determined based on the sensor data in order to generate a heat map. One or more of the plurality of predicted trajectories may be compared to the heat map. The vehicle may be controlled in the autonomous driving mode based on the comparison.

Verifying predicted trajectories using a grid-based approach

An autonomous vehicle includes sensor subsystem(s) that output a sensor signal. A perception subsystem (i) detects an agent in a vicinity of the autonomous vehicle and (ii) generates a motion signal that describes at least one of a past motion or a present motion of the agent. An intention prediction subsystem processes the sensor signal to generate an intention signal that describes at least one intended action of the agent. A behavior prediction subsystem processes the motion signal and the intention signal to generate a behavior prediction signal that describes at least one predicted behavior of the agent. A planner subsystem processes the behavior prediction signal to plan a driving decision for the autonomous vehicle.

Predicting behaviors of road agents using intermediate intention signals

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for intervention behavior prediction. One of the methods includes receiving data characterizing a scene that includes a first agent and a second agent in an environment. A confounder prediction input generated from the data is processed using a confounder prediction model. A plurality of predicted conditional probability distributions is generated, wherein each predicted conditional probability distribution is conditioned on: (i) a planned intervention by the second agent, and (ii) the confounder variable belonging to a corresponding confounder class. An intervention behavior prediction for the first agent is generated based on the plurality of the predicted conditional probability distributions and the confounder distribution, wherein the intervention behavior prediction includes a probability distribution over the plurality of the possible behaviors for the first agent in reaction to the second agent performing the planned intervention.

Intervention behavior prediction

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Ranking agents near autonomous vehicles by mutual importance

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining occupancies of surrounding agents. One of the methods includes obtaining scene data characterizing an environment at a current time point; processing a first network input generated from the scene data using a first neural network to generate an intermediate output; obtaining an identification of a future time point that is after the current time point; and generating, from the intermediate output and the future time point, an occupancy output, wherein the occupancy output comprises respective occupancy probabilities for each of a plurality of locations in the environment, wherein the respective occupancy probability for each location characterizes a likelihood that one or more agents will occupy the location at the future time point.

Predicting occupancy probabilities of surrounding agents

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating training data for training a machine learning model to perform trajectory prediction. One of the methods includes: obtaining a training input, the training input including (i) data characterizing an agent in an environment as of a first time and (ii) data characterizing a candidate trajectory of the agent in the environment for a first time period that is after the first time. A long-term label for the candidate trajectory that indicates whether the agent actually followed the candidate trajectory for the first time period is determined. A short-term label for the candidate trajectory that indicates whether the agent intended to follow the candidate trajectory is determined. A ground-truth probability for the candidate trajectory is determined. The training input is associated with the ground-truth probability for the candidate trajectory in the training data.

Generating trajectory labels from short-term intention and long-term result

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating candidate future trajectories for agents. One of the methods includes obtaining scene data characterizing a scene in an environment at a current time point; for each of a plurality of lane segments, processing a model input comprising (i) features of the lane segment and (ii) features of the target agent using a machine learning model that is configured to process the model input to generate a respective score for the lane segment that represents a likelihood that the lane segment will be a first lane segment traversed by the target agent after the current time point; selecting, as a set of seed lane segments, a proper subset of the plurality of lane segments based on the respective scores; and generating a plurality of candidate future trajectories for the target agent.

Growing community of inventors

Mountain View, CA, United States of America

Khaled Refaat