Uatc, LLC

Aurora Operations, Inc.

Mitsubishi Electric Research Laboratories, Inc.

Waabi Innovation Inc.

Raquel Urtasun

Shenlong Wang

Bin Yang

Ming Liang

Wei-Chiu Ma

Sergio Casas

Sivabalan Manivasagam

Wenyuan Zeng

Mengye Ren

Min Bai

Shun Da Suo

Renjie Liao

Wenjie Luo

Kelvin Ka Wing Wong

Namdar Homayounfar

Cole Christian Gulino

Abbas Sadat

Davi Eugenio Nascimento Frossard

Xuanyuan Tu

Gellert Sandor Mattyus

Julieta Martinez Covarrubias

Jerry Junkai Liu

Yuwen Xiong

Eric Randall Kee

Ersin Yumer

Andrei Pokrovsky

Sean Segal

Ioan Andrei Barsan

Justin Liang

Yun Chen

Xinchen Yan

Hongbo Fan

Tsun-Hsuan Wang

Shivam Duggal

Jingkang Wang

Xinkai Wei

Shrindihi Kowshika Lakshmikanth

Frieda Rong

Chris Jia-Han Zhang

Katie Z Luo

Lingyun Li

Alexander Yuhao Cui

Rui Hu

Yuichi Taguchi

Srikumar Ramalingam

Ioan Andrei Bârsan

Yen-Chen Lin

Nicholas Baskar Vadivelu

Ajay Jain

Jashan Sunil Shewakramani

Runsheng Benson Guo

Bob Qingyuan Wei

Song Feng

Runsheng Guo

Adam Wlodzimierz Harley

Mehmet Ersin Yumer

Ricson Cheng

Shuhan Tan

Yushu Huang

Sourav Biswas

Can Cui

Ting Wei Liu

Qiang Zhang

Shrinidhi Kowshika Lakshmikanth

Michel Antunes

Tsun-hsuan Wang

Alexandre Doubov

Ze Yang

Chris Jia Han Zhang

Shrinidihi Kowshika Lakshmikanth

Ava Alison Pun

Sebastián David Regalado Lozano

Anqi Joyce Yang

Generally, the disclosed systems and methods implement improved detection of objects in three-dimensional (3D) space. More particularly, an improved 3D object detection system can exploit continuous fusion of multiple sensors and/or integrated geographic prior map data to enhance effectiveness and robustness of object detection in applications such as autonomous driving. In some implementations, geographic prior data (e.g., geometric ground and/or semantic road features) can be exploited to enhance three-dimensional object detection for autonomous vehicle applications. In some implementations, object detection systems and methods can be improved based on dynamic utilization of multiple sensor modalities. More particularly, an improved 3D object detection system can exploit both LIDAR systems and cameras to perform very accurate localization of objects within three-dimensional space relative to an autonomous vehicle. For example, multi-sensor fusion can be implemented via continuous convolutions to fuse image data samples and LIDAR feature maps at different levels of resolution.

Three-dimensional object detection

Systems and methods for determining object intentions through visual attributes are provided. A method can include determining, by a computing system, one or more regions of interest. The regions of interest can be associated with surrounding environment of a first vehicle. The method can include determining, by a computing system, spatial features and temporal features associated with the regions of interest. The spatial features can be indicative of a vehicle orientation associated with a vehicle of interest. The temporal features can be indicative of a semantic state associated with signal lights of the vehicle of interest. The method can include determining, by the computing system, a vehicle intention. The vehicle intention can be based on the spatial and temporal features. The method can include initiating, by the computing system, an action. The action can be based on the vehicle intention.

System and method for determining object intention through visual attributes

A computing system can include one or more processors and one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the computing system to perform operations including obtaining model structure data indicative of a plurality of parameters of a machine-learned model; determining a codebook comprising a plurality of centroids, the plurality of centroids having a respective index of a plurality of indices indicative of an ordering of the codebook; determining a plurality of codes respective to the plurality of parameters, the plurality of codes respectively comprising a code index of the plurality of indices corresponding to a closest centroid of the plurality of centroids to a respective parameter of the plurality of parameters; and providing encoded data as an encoded representation of the plurality of parameters of the machine-learned model, the encoded data comprising the codebook and the plurality of codes.

Compression of machine-learned models by vector quantization

Systems and methods for detecting and tracking objects are provided. In one example, a computer-implemented method includes receiving sensor data from one or more sensors. The method includes inputting the sensor data to one or more machine-learned models including one or more first neural networks configured to detect one or more objects based at least in part on the sensor data and one or more second neural networks configured to track the one or more objects over a sequence of sensor data. The method includes generating, as an output of the one or more first neural networks, a 3D bounding box and detection score for a plurality of object detections. The method includes generating, as an output of the one or more second neural networks, a matching score associated with pairs of object detections. The method includes determining a trajectory for each object detection.

End-to-end tracking of objects

Systems and methods for generating simulation data based on real-world environments are provided. A method includes obtaining multi-modal sensor data indicative of a dynamic object within an environment of a robotic platform. The multi-modal sensor data is associated with a plurality of timesteps including a first timestep and a second timestep. The method includes providing the multi-modal sensor data indicative of the dynamic object within the environment as an input to a machine-learned dynamic object removal model. And, the method includes receiving as an output of the machine-learned dynamic object removal model, in response to receipt of the multi-modal sensor data, a scene representation indicative of at least a portion of the environment including a reconstructed region based at least in part on removal of the dynamic object and multiple levels of granularity. The scene representation is used as a template for generating different simulations within the depicted environment.

Systems and methods for dynamic object removal from three-dimensional data

Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same are disclosed. The computing system can include an object detection model and a graph neural network including a plurality of nodes and a plurality of edges. The computing system can be configured to input sensor data into the object detection model; receive object detection data describing the location of the plurality of the actors relative to the autonomous vehicle as an output of the object detection model; input the object detection data into the graph neural network; iteratively update a plurality of node states respectively associated with the plurality of nodes; and receive, as an output of the graph neural network, the motion forecast data with respect to the plurality of actors.

Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same

Trajectory value learning for autonomous systems includes generating an environment image from sensor input and processing the environment image through an image neural network to obtain a feature map. Trajectory value learning further includes sampling possible trajectories to obtain a candidate trajectory for an autonomous system, extracting, from the feature map, feature vectors corresponding to the candidate trajectory, combining the feature vectors into the input vector, and processing, by a score neural network model, the input vector to obtain a projected score for the candidate trajectory. Trajectory value learning further includes selecting, from the candidate trajectories, the candidate trajectory as a selected trajectory based on the projected score, and implementing the selected trajectory.

Trajectory value learning for autonomous systems

Systems and methods for determining a location based on image data are provided. A method can include receiving, by a computing system, a query image depicting a surrounding environment of a vehicle. The query image can be input into a machine-learned image embedding model and a machine-learned feature extraction model to obtain a query embedding and a query feature representation, respectively. The method can include identifying a subset of candidate embeddings that have embeddings similar to the query embedding. The method can include obtaining a respective feature representation for each image associated with the subset of candidate embeddings. The method can include determining a set of relative displacements between each image associated with the subset of candidate embeddings and the query image and determining a localized state of a vehicle based at least in part on the set of relative displacements.

Image based localization system

A computer-implemented method for determining scene-consistent motion forecasts from sensor data can include obtaining scene data including one or more actor features. The computer-implemented method can include providing the scene data to a latent prior model, the latent prior model configured to generate scene latent data in response to receipt of scene data, the scene latent data including one or more latent variables. The computer-implemented method can include obtaining the scene latent data from the latent prior model. The computer-implemented method can include sampling latent sample data from the scene latent data. The computer-implemented method can include providing the latent sample data to a decoder model, the decoder model configured to decode the latent sample data into a motion forecast including one or more predicted trajectories of the one or more actor features. The computer-implemented method can include receiving the motion forecast including one or more predicted trajectories of the one or more actor features from the decoder model.

Systems and methods for latent distribution modeling for scene-consistent motion forecasting

Systems and methods for streaming sensor packets in real-time are provided. An example method includes obtaining a sensor data packet representing a first portion of a three-hundred and sixty degree view of a surrounding environment of a robotic platform. The method includes generating, using machine-learned model(s), a local feature map based at least in part on the sensor data packet. The local feature map is indicative of local feature(s) associated with the first portion of the three-hundred and sixty degree view. The method includes updating, based at least in part on the local feature map, a spatial map to include the local feature(s). The spatial map includes previously extracted local features associated with a previous sensor data packet representing a different portion of the three-hundred and sixty degree view than the first portion. The method includes determining an object within the surrounding environment based on the updated spatial map.

Systems and methods for sensor data packet processing and spatial memory updating for robotic platforms

Systems and methods are disclosed for motion forecasting and planning for autonomous vehicles. For example, a plurality of future traffic scenarios are determined by modeling a joint distribution of actor trajectories for a plurality of actors, as opposed to an approach that models actors individually. As another example, a diversity objective is evaluated that rewards sampling of the future traffic scenarios that require distinct reactions from the autonomous vehicle. An estimated probability for the plurality of future traffic scenarios can be determined and used to generate a contingency plan for motion of the autonomous vehicle. The contingency plan can include at least one initial short-term trajectory intended for immediate action of the AV and a plurality of subsequent long-term trajectories associated with the plurality of future traffic scenarios.

Systems and methods for motion forecasting and planning for autonomous vehicles

Generally, the disclosed systems and methods utilize multi-task machine-learned models for object intention determination in autonomous driving applications. For example, a computing system can receive sensor data obtained relative to an autonomous vehicle and map data associated with a surrounding geographic environment of the autonomous vehicle. The sensor data and map data can be provided as input to a machine-learned intent model. The computing system can receive a jointly determined prediction from the machine-learned intent model for multiple outputs including at least one detection output indicative of one or more objects detected within the surrounding environment of the autonomous vehicle, a first corresponding forecasting output descriptive of a trajectory indicative of an expected path of the one or more objects towards a goal location, and/or a second corresponding forecasting output descriptive of a discrete behavior intention determined from a predefined group of possible behavior intentions.

Multi-task machine-learned models for object intention determination in autonomous driving

Systems and methods for generating attention masks are provided. In particular, a computing system can access sensor data and map data for an area around an autonomous vehicle. The computing system can generate a voxel grid representation of the sensor data and map data. The computing system can generate an attention mask based on the voxel grid representation. The computing system can generate, by using the voxel grid representation and the attention mask as input to a machine-learned model, an attention weighted feature map. The computing system can determine using the attention weighted feature map, a planning cost volume for an area around the autonomous vehicle. The computing system can select a trajectory for the autonomous vehicle based, at least in part, on the planning cost volume.

Systems and methods for using attention masks to improve motion planning

Systems and methods for identifying travel way features in real time are provided. A method can include receiving two-dimensional and three-dimensional data associated with the surrounding environment of a vehicle. The method can include providing the two-dimensional data as one or more input into a machine-learned segmentation model to output a two-dimensional segmentation. The method can include fusing the two-dimensional segmentation with the three-dimensional data to generate a three-dimensional segmentation. The method can include storing the three-dimensional segmentation in a classification database with data indicative of one or more previously generated three-dimensional segmentations. The method can include providing one or more datapoint sets from the classification database as one or more inputs into a machine-learned enhancing model to obtain an enhanced three-dimensional segmentation. And, the method can include identifying one or more travel way features based at least in part on the enhanced three-dimensional segmentation.

System and method for identifying travel way features for autonomous vehicle motion control

Systems and methods for performing semantic segmentation of three-dimensional data are provided. In one example embodiment, a computing system can be configured to obtain sensor data including three-dimensional data associated with an environment. The three-dimensional data can include a plurality of points and can be associated with one or more times. The computing system can be configured to determine data indicative of a two-dimensional voxel representation associated with the environment based at least in part on the three-dimensional data. The computing system can be configured to determine a classification for each point of the plurality of points within the three-dimensional data based at least in part on the two-dimensional voxel representation associated with the environment and a machine-learned semantic segmentation model. The computing system can be configured to initiate one or more actions based at least in part on the per-point classifications.

Semantic segmentation of three-dimensional data

Growing community of inventors

Toronto, Canada

Raquel Urtasun