Toyota Research Institute, Inc.

Leland Stanford Junior University

Toyota Jidosha Kabushiki Kaisha

University of California

Massachusetts Institute of Technology

Toyota Technological Institute at Chicago

Woven Planet North America, Inc.

The Board of Trustees of the Leland Standford Junior Univeristy

Adrien David Gaidon

Vitor Guizilini

Rares A Ambrus

Sudeep Pillai

Jie Li

Igor Vasiljevic

Kuan-Hui Lee

Dennis Park

Sergey Zakharov

Jiexiong Tang

Gregory Shakhnarovich

Arjun Bhargava

Wadim Kehl

Hanme Kim

Ehsan Adeli-Mosabbeb

Mac Schwager

Boris Ivanovic

Victor Vaquero Gomez

Karttikeya Mangalam

Pavel V Tokmakov

Guy Rosman

Leonidas John Guibas

German Ros Sanchez

Blake Warren Wulfe

Wolfram Burgard

Juan Carlos Niebles Duque

Jia-En M Pan

Marco Pavone

Or Litany

Juan Carlos Niebles

Negar Zahedi Mehr

Rui Hou

Haruki Nishimura

Felipe Codevilla

Eder Santana

James Joseph Kuffner

Alexander Alspach

Sergey Levine

Nikolaos Michalakis

Kun-Hsin Chen

Naveen Suresh Kuppuswamy

Andrew M Beaulieu

Fredo Durand

Matthew T Kliemann

Logan Michael Ellis

Takaaki Tagawa

Xipeng Wang

Haofeng Chen

Vincent Sitzmann

Bertrand Douillard

Allan Jabri

Jiading Fang

Matthew R Walter

Kuan-hui Lee

Jean Mercat

Allan Ricardo Raventos Knohr

Joshua Tenenbaum

German Ros

Allan Raventos

Nishant Rai

Ehsan Adeli Mosabbeb

Dian Chen

Jeff Walls, Sr

Rowan Thomas Mcallister

Shyamal D Buch

Vitor Campagnolo Guizilini

Yue Wang

Jiajun Wu

De-An Huang

Yu Yao

Haoye Cai

Boxiao Pan

Amine Elhafsi

Andreas Buehler

Mingyu Wang

Kuan Lee

Bingbin Liu

Zhangjie Cao

Xiangru Huang

Justin Solomon

A method of representation learning for object detection from unlabeled point cloud sequences is described. The method includes detecting moving object traces from temporally-ordered, unlabeled point cloud sequences. The method also includes extracting a set of moving objects based on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences. The method further includes classifying the set of moving objects extracted from on the moving object traces detected from the sequence of temporally-ordered, unlabeled point cloud sequences. The method also includes estimating 3D bounding boxes for the set of moving objects based on the classifying of the set of moving objects.

Representation learning for object detection from unlabeled point cloud sequences

A method for improving 3D object detection via object-level augmentations is described. The method includes recognizing, using an image recognition model of a differentiable data generation pipeline, an object in an image of a scene. The method also includes generating, using a 3D reconstruction model, a 3D reconstruction of the scene from the image including the recognized object. The method further includes manipulating, using an object level augmentation model, a random property of the object by a random magnitude at an object level to determine a set of properties and a set of magnitudes of an object manipulation that maximizes a loss function of the image recognition model. The method also includes training a downstream task network based on a set of training data generated based on the set of properties and the set of magnitudes of the object manipulation, such that the loss function is minimized.

Adversarial object-aware neural scene rendering for 3D object detection

A method includes receiving first driving data associated with a first vehicle, receiving second driving data associated with one or more vehicles around the first vehicle, creating training data by labeling the first driving data as positive data and treating the second driving data as unlabeled, and using the training data to train a classifier to predict whether driving data input to the classifier is positive or unlabeled.

Method and system for learning reward functions for driving using positive-unlabeled reward learning

Systems and methods for self-supervised learning for visual odometry using camera images, may include: estimating correspondences between keypoints of a target camera image and keypoints of a context camera image; based on the keypoint correspondences, lifting a set of 2D keypoints to 3D, using a neural camera model; and projecting the 3D keypoints into the context camera image using the neural camera model. Some embodiments may use the neural camera model to achieve the lifting and projecting of keypoints without a known or calibrated camera model.

Systems and methods for generic visual odometry using learned features via neural camera models

A method for 3D object detection is described. The method includes concurrently training a monocular depth network and a 3D object detection network. The method also includes predicting, using a trained monocular depth network, a monocular depth map of a monocular image of a video stream. The method further includes inferring a 3D point cloud of a 3D object within the monocular image according to the predicted monocular depth map. The method also includes predicting 3D bounding boxes from a selection of 3D points from the 3D point cloud of the 3D object based on a selection regression loss.

Monocular object detection via end-to-end differentiable pipeline

System, methods, and other embodiments described herein relate to training a depth model for monocular depth estimation using photometric loss masks derived from motion estimates of dynamic objects. In one embodiment, a method includes generating depth maps from images of an environment. The method includes determining motion of points within the depth maps. The method includes associating the points between the depth maps to identify an object according to a correlation of the motion for a first cluster of the points with a second cluster of the points. The method includes providing the depth maps and the object as an electronic output.

Self-supervised monocular depth estimation via rigid-motion embeddings

A method for controlling an agent to navigate through an environment includes generating a depth map associated with a monocular image of the environment. The method also includes generating a group of surface normal. Each surface normal of the group of surface normals is associated with a respective polygon of a group of polygons associated with the depth map. The method further includes identifying one or more ground planes in the depth map based on the group of surface normal. The method further includes controlling the agent to navigate through the environment based on identifying the one or more ground planes.

System and method for self-supervised monocular ground-plane extraction

Systems, methods, and other embodiments described herein relate to estimating scaled depth maps by sampling variational representations of an image using a learning model. In one embodiment, a method includes encoding data embeddings by a learning model to form conditioned latent representations using attention networks, the data embeddings including features about an image from a camera and calibration information about the camera. The method also includes computing a probability distribution of the conditioned latent representations by factoring scale priors. The method also includes sampling the probability distribution to generate variations for the data embeddings. The method also includes estimating scaled depth maps of a scene from the variations at different coordinates using the attention networks.

Systems and methods for estimating scaled maps by sampling representations from a learning model

A method for depth estimation performed by a depth estimation system associated with an agent includes determining a first depth of a first image and a second depth of a second image, the first image and the second image being captured by a sensor associated with the agent. The method also includes generating a first 3D image of the first image based on the first depth, a first pose associated with the sensor, and the second image. The method further includes generating a warped depth image based on transforming the first depth in accordance with the first pose. The method also includes updating the first pose based on a second pose associated with the warped depth image and the second depth, and updating the first 3D image based on the updated first pose. The method further includes controlling an action of the agent based on the updated first 3D image.

Depth estimation based on ego-motion estimation and residual flow estimation

Systems and methods for multi-view cooperative contrastive self-supervised learning, may include receiving a plurality of video sequences, the video sequences comprising a plurality of image frames; applying selected images of a first and second video sequence of the plurality of video sequences to a plurality of different encoders to derive a plurality of embeddings for different views of the selected images of the first and second video sequences; determining distances of the derived plurality of embeddings for the selected images of the first and second video sequences; detecting inconsistencies in the determined distances; and predicting semantics of a future image based on the determined distances.

Cooperative-contrastive learning systems and methods

A system for producing a depth map can include a processor and a memory. The memory can store a neural network. The neural network can include an encoding portion module, a multi-frame feature matching portion module, and a decoding portion module. The encoding portion module can include instructions that, when executed by the processor, cause the processor to encode an image to produce single-frame features. The multi-frame feature matching portion module can include instructions that, when executed by the processor, cause the processor to process the single-frame features to produce information. The decoding portion module can include instructions that, when executed by the processor, cause the processor to decode the information to produce the depth map. A first training dataset, used to train the multi-frame feature matching portion module, can be different from a second training dataset used to train the encoding portion module and the decoding portion module.

Producing a depth map from a monocular two-dimensional image

A method for self-calibrating alignment between image data and point cloud data utilizing a machine learning model includes receiving, with an electronic control unit, image data from a vision sensor and point cloud data from a depth sensor, implementing, with the electronic control unit, a machine learning model trained to: align the point cloud data and the image data based on a current calibration, detect a difference in alignment of the point cloud data and the image data, adjust the current calibration based on the difference in alignment, and output a calibrated embedding feature map based on adjustments to the current calibration.

Self-calibration for decoration based sensor fusion method

A method for tracking occluded objects includes encoding locations of a plurality of objects in an environment, determining a target object, receiving a first end point corresponding to a position of the target object before occlusion behind an occlusion object, distributing a hypothesis between both sides of the occlusion object during occlusion from a subsequent frame of the sequence of frames, receiving a second end point corresponding to a position of the target object after emerging from occlusion from another subsequent frame of the sequence of frames, and determining a trajectory of the target object when occluded by the occlusion object by performing inferences using a spatio-temporal probabilistic graph based on the current frame and the subsequent frames of the sequence of frames. The trajectory of the target object when occluded is used as a learning model for future target objects that are occluded by the occlusion object.

Systems and methods for tracking occluded objects

A method for 3D object detection is described. The method includes predicting, using a trained monocular depth network, an estimated monocular input depth map of a monocular image of a video stream and an estimated depth uncertainty map associated with the estimated monocular input depth map. The method also includes feeding back a depth uncertainty regression loss associated with the estimated monocular input depth map during training of the trained monocular depth network to update the estimated monocular input depth map. The method further includes detecting 3D objects from a 3D point cloud computed from the estimated monocular input depth map based on seed positions selected from the 3D point cloud and the estimated depth uncertainty map. The method also includes selecting 3D bounding boxes of the 3D objects detected from the 3D point cloud based on the seed positions and an aggregated depth uncertainty.

Systems and methods for uncertainty aware monocular 3D object detection

A method for keypoint matching performed by a semantically aware keypoint matching model includes generating a semanticly segmented image from an image captured by a sensor of an agent, the semanticly segmented image associating a respective semantic label with each pixel of a group of pixels associated with the image. The method also includes generating a set of augmented keypoint descriptors by augmenting, for each keypoint of the set of keypoints associated with the image, a keypoint descriptor with semantic information associated with one or more pixels, of the semantically segmented image, corresponding to the keypoint. The method further includes controlling an action of the agent in accordance with identifying a target image having one or more first augmented keypoint descriptors that match one or more second augmented keypoint descriptors of the set of augmented keypoint descriptors.

Semantically aware keypoint matching

Growing community of inventors

Mountain View, CA, United States of America

Adrien David Gaidon