Waymo LLC

Kai Ding

Khaled Refaat

Stephane Ross

Vishu Goyal

Stéphane Ross

Wei Chai

Haoyu Chen

Minfa Wang

Jared Stephen Russell

Aishwarya Parasuram

Xin Liu

Alisha Saxena

Anoosha Sagar

Shubham Gupta

Benjamin Sapp

Maher Mneimneh

Tacettin Dogacan Guney

Andre Liang Cornman

Yang Wang

Olivier Gravel Aubin

Avikalp Srivastava

Hee Jin Jeong

Xiaoting Yin

Chenjie Yang

Aspects of the disclosure provide methods of modeling wrong-way driving of road users. For instance, log data including an observed trajectory of a first road user may be accessed. A first set of candidate lane segments for wrong-way driving may be identified from map information. A second set of candidate lane segments for not wrong-way driving may be identified from the map information. For each candidate lane segment in the first set and in the second set, a distance cost between the candidate lane segment and the observed trajectory may be determined. A candidate lane segment may be selected from at least one of the first set or the second set based on the determined distance costs. The selected candidate lane segment may be used to train a model to provide a likelihood of a second road user being engaged in wrong-way driving in a lane.

Wrong-way driving modeling

To operate an autonomous vehicle, a rail agent is detected in a vicinity of the autonomous vehicle using a detection system. One or more tracks are determined on which the detected rail agent is possibly traveling, and possible paths for the rail agent are predicted based on the determined one or more tracks. One or more motion paths are determined for one or more probable paths from the possible paths, and a likelihood for each of the one or more probable paths is determined based on each motion plan. A path for the autonomous vehicle is then determined based on a most probable path associated with a highest likelihood for the rail agent, and the autonomous vehicle is operated using the determined path.

Behavior prediction for railway agents for autonomous driving system

The technology relates to controlling a vehicle in an autonomous driving mode in accordance with behavior predictions for other road users in the vehicle's vicinity. In particular, the vehicle's onboard computing system may predict whether another road user will perform a 'continuing' lane driving operation, such as going straight in a turn-only lane. Sensor data from detected/observed objects in the vehicle's nearby environment may be evaluated in view of one or more possible behaviors for different types of objects. In addition, roadway features, in particular whether lane segments are connected in a roadgraph, are also evaluated to determine probabilities of whether other road users may make an improper continuing lane driving operation. This is used to generate more accurate behavior predictions, which the vehicle can use to take alternative (e.g., corrective) driving actions.

Continuing lane driving prediction

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using candidate future intents.

Agent trajectory prediction using dynamic candidate future intents

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing knowledge distillation for autonomous vehicles. One of the methods includes obtaining sensor data characterizing an environment, wherein the sensor data has been captured by one or more sensors on-board a vehicle in the environment; processing, for each of one or more surrounding agents in the environment, a network input generated from the sensor data using a neural network to generate an agent discomfort prediction that characterizes a level of discomfort of the agent; combining the one or more agent discomfort predictions to generate an aggregated discomfort score; and providing the aggregated discomfort score to a path planning system of the vehicle in order to generate a future path of the vehicle.

Knowledge distillation for autonomous vehicles

Aspects of the disclosure provide for controlling a vehicle in an autonomous driving mode. For instance, sensor data for an object as well as a plurality of predicted trajectories may be received. Each predicted trajectory may represent a plurality of possible future locations for the object. A grid including a plurality of cells, each being associated with a geographic area, may be generated. Probabilities that the object will enter the geographic area associated with each of the plurality of cells over a period of time into the future may be determined based on the sensor data in order to generate a heat map. One or more of the plurality of predicted trajectories may be compared to the heat map. The vehicle may be controlled in the autonomous driving mode based on the comparison.

Verifying predicted trajectories using a grid-based approach

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that obtain scene features in an environment that includes an autonomous vehicle, a first target agent, and a second target agent, and determines whether the first target agent is an emergency vehicle that is active at a current time point. In response to determining that the first target agent is an emergency vehicle that is active at the current time point, an input is generated from the scene features. The input can characterize the scene and indicate that the first target agent is an emergency vehicle that is active at the current time point. Also in response, the input can be processed using a machine learning model that is configured to generate a trajectory prediction output for the second target agent that characterizes predicted future behavior of the second target agent after the current time point.

Predicting agent trajectories in the presence of active emergency vehicles

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting near-curb driving behavior. One of the methods includes obtaining agent trajectory data for an agent in an environment, the agent trajectory data comprising a current location and current values for a predetermined set of motion parameters of the agent; processing a model input generated from the agent trajectory data using a trained machine learning model to generate a model output comprising a prediction of whether the agent will exhibit near-curb driving behavior within a predetermined timeframe, wherein an agent exhibits near-curb driving behavior when the agent operates within a particular distance of an edge of a road in the environment; and using the prediction to generate a planned path for a vehicle in the environment.

Predicting near-curb driving behavior on autonomous vehicles

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining a task schedule for generating prediction data for different agents. In one aspect, a method comprises receiving data that characterizes an environment in a vicinity of a vehicle at a current time step, the environment comprising a plurality of agents; receiving data that identifies high-priority agents for which respective data characterizing the agents must be generated at the current time step; identifying available computing resources at the current time step; processing the data that characterizes the environment using a complexity scoring model to determine a respective complexity score for each of the high-priority agents; and determining a schedule for the current time step that allocates the generation of the data characterizing the high-priority agents across the available computing resources based on the complexity scores.

Task scheduling for agent prediction

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for identifying high-priority agents in the vicinity of a vehicle. The high-priority agents can be identified based on a set of mutual importance scores in which each mutual importance score indicates an estimated mutual relevance between the vehicle and a different agent from a set of agents on planning decisions of the other. The mutual importance scores can be calculated based on importance scores assessed from the perspectives of both the vehicle and the agents.

Ranking agents near autonomous vehicles by mutual importance

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that obtain scene features in an environment that includes an autonomous vehicle and a target agent at a current time point; determine whether the target agent is an emergency vehicle that is active at the current time point; generate from the scene features, an input (i) that includes the scene features and (ii) that indicates whether the target vehicle is an emergency vehicle that is active at the current time point; and process the input using a machine learning model that is configured to generate a behavior prediction output for the target agent that characterizes predicted future behavior of the target agent after the current time point.

Behavior predictions for active emergency vehicles

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining occupancies of surrounding agents. One of the methods includes obtaining scene data characterizing an environment at a current time point; processing a first network input generated from the scene data using a first neural network to generate an intermediate output; obtaining an identification of a future time point that is after the current time point; and generating, from the intermediate output and the future time point, an occupancy output, wherein the occupancy output comprises respective occupancy probabilities for each of a plurality of locations in the environment, wherein the respective occupancy probability for each location characterizes a likelihood that one or more agents will occupy the location at the future time point.

Predicting occupancy probabilities of surrounding agents

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating candidate future trajectories for agents. One of the methods includes obtaining scene data characterizing a scene in an environment at a current time point; for each of a plurality of lane segments, processing a model input comprising (i) features of the lane segment and (ii) features of the target agent using a machine learning model that is configured to process the model input to generate a respective score for the lane segment that represents a likelihood that the lane segment will be a first lane segment traversed by the target agent after the current time point; selecting, as a set of seed lane segments, a proper subset of the plurality of lane segments based on the respective scores; and generating a plurality of candidate future trajectories for the target agent.

Growing community of inventors

Santa Clara, CA, United States of America

Kai Ding