Waymo LLC

Sri International

Mayank Bansal

Harpreet Singh Sawhney

Jayakrishnan Kumar Eledath

Dragomir Dimitrov Anguelov

Hui Ting Cheng

Ajay Divakaran

Bogdan C Matei

Timothy J Shields

Xutao Lv

Chaitanya Desai

Abhijit S Ogale

Rakesh Kumar

Theodore Armand Camus

Alexander Krizhevsky

Benjamin Sapp

Yuning Chai

John Benjamin Southall

Supun Samarasekera

Girish Acharya

Michael John Wolverton

John R Fields

Bogdan Calin Mihai Matei

Gary Alan Greene

Sang-Hack Jung

Raia Hadsell

Jayan Eledath

Aveek Kumar Das

Aaron Spaulding

Jinkyu Kim

Aparna Taneja

Ankit Kumar

Aastha Jain

Manish Kumar

Ron Krakower

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using anchor trajectories.

Agent trajectory prediction using anchor trajectories

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting occupancies of agents. One of the methods includes obtaining scene data characterizing a current scene in an environment; and processing a neural network input comprising the scene data using a neural network to generate a neural network output, wherein: the neural network output comprises respective occupancy outputs corresponding to a plurality of agent types at one or more future time points; the occupancy output for each agent type at a first future time point comprises respective occupancy probabilities for a plurality of locations in the environment; and in the occupancy output for each agent type at the first future time point, the respective occupancy probability for each location characterizes a likelihood that an agent of the agent type will occupy the location at the first future time point.

Behavior prediction of surrounding agents

Aspects of the disclosure relate to verifying the location of an object of a particular type. For instance, a plurality of images of an environment of the vehicle may be received. Associated objects of the particular type may be identified in ones of the plurality of images. A plurality of estimated locations may be determined for the object using a plurality of different localization techniques. For each image of the ones of the plurality of images, determine a reprojection error for each of the plurality of estimated locations. For each of the plurality of estimated locations, an error score is determined based on the reprojection errors. An estimated location may be selected from the plurality of estimated locations based on the determined error score. This selected location may be used to control a vehicle in an autonomous driving mode.

Object localization for autonomous driving by visual tracking and image reprojection

Aspects of the disclosure relate to detecting an emergency vehicle. For instance, a plurality of images may be taken from a perspective of an autonomous vehicle. One or more gates representing a region of interest at a respective distance from the vehicle may be generated for the images. A plurality of lights may be detected within the one or more gates. A first candidate emergency vehicle may be identified from a detected plurality of lights in one or more gates of one of the images, and a second candidate emergency vehicle may be identified from a detected plurality of lights in one or more gates of another of the images. The first and second candidate emergency vehicles are determined to be the same emergency vehicle and to be active. An operational system of the autonomous vehicle is controlled based on the determination that the given emergency vehicle is active.

Detection of emergency vehicles

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for planning a trajectory of a vehicle. One of the methods includes obtaining input data for planning a driving trajectory for a vehicle, the input data comprising an intended route for the vehicle and data characterizing an environment in a vicinity of the vehicle; processing the input data using an input encoder neural network to generate feature data that includes a respective feature representation for each of a plurality of locations in the environment; applying spatial attention to the feature representations to generate a respective attention weight for each of the plurality of locations; generating a respective attended feature representation for each of the plurality of locations; generating a bottlenecked representation of the attended feature representations; and generating a planned future trajectory from at least the bottlenecked representation.

Neural networks with attention al bottlenecks for trajectory planning

A computing system includes a vision-based user interface platform to, among other things, analyze multi-modal user interactions, semantically correlate stored knowledge with visual features of a scene depicted in a video, determine relationships between different features of the scene, and selectively display virtual elements on the video depiction of the scene. The analysis of user interactions can be used to filter the information retrieval and correlating of the visual features with the stored knowledge.

Real-time human-machine collaboration using big data driven augmented reality technologies

Systems, methods, devices, and other techniques for training a trajectory planning neural network system to determine waypoints for trajectories of vehicles. A neural network training system can train the trajectory planning neural network system on the multiple training data sets. Each training data set can include: (i) a first training input that characterizes a set of waypoints that represent respective locations of a vehicle at each of a series of first time steps, (ii) a second training input that characterizes at least one of (a) environmental data that represents a current state of an environment of the vehicle or (b) navigation data that represents a planned navigation route for the vehicle, and (iii) a target output characterizing a waypoint that represents a target location of the vehicle at a second time step that follows the series of first time steps.

Neural networks for vehicle trajectory planning

A computing system and method for identifying related visual content of a collection of visual media files comprising one or more image files and video files includes monitoring inputs to the computing system, the inputs associated with a user interaction with electronic content using the computer system, identifying a visual media file in the collection of visual media files relevant to the electronic content based on a semantic label assigned to the visual media file by the computing system, creating a representative image of the identified visual media file, and displaying the representative image for selection. The computing system enables a selection of the displayed representative image for association of the identified visual media file with the electronic content.

Multi-dimensional realization of visual content of an image collection

Systems, methods, devices, and other techniques for planning a trajectory of a vehicle. A computing system can implement a trajectory planning neural network configured to, at each time step of multiple time steps: obtain a first neural network input and a second neural network input. The first neural network input can characterize a set of waypoints indicated by the waypoint data, and the second neural network input can characterize (a) environmental data that represents a current state of an environment of the vehicle and (b) navigation data that represents a planned navigation route for the vehicle. The trajectory planning neural network may process the first neural network input and the second neural network input to generate a set of output scores, where each output score in the set of output scores corresponds to a different location of a set of possible locations in a vicinity of the vehicle.

A computing system for realizing visual content of an image collection executes feature detection algorithms and semantic reasoning techniques on the images in the collection to elicit a number of different types of visual features of the images. The computing system indexes the visual features and provides technologies for multi-dimensional content-based clustering, searching, and iterative exploration of the image collection using the visual features and/or the visual feature indices.

Methods and apparatuses of the present invention generally relate to generating actionable data based on multimodal data from unsynchronized data sources. In an exemplary embodiment, the method comprises receiving multimodal data from one or more unsynchronized data sources, extracting concepts from the multimodal data, the concepts comprising at least one of objects, actions, scenes and emotions, indexing the concepts for searchability; and generating actionable data based on the concepts.

Method and apparatus for correlating and viewing disparate data

Growing community of inventors

Plainsboro, NJ, United States of America

Mayank Bansal