Toyota Research Institute, Inc.

Toyota Jidosha Kabushiki Kaisha

Charles Christopher Ochoa

Kuan-Hui Lee

Chao Fang

Kun-Hsin Chen

Arjun Bhargava

Vitor Guizilini

Jie Li

A method for vehicle prediction, planning, and control is described. The method includes separately encoding traffic state information at an intersection into corresponding traffic state latent spaces. The method also includes aggregating the corresponding traffic state latent spaces to form a generalized traffic geometry latent space. The method further includes interpreting the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories. The method also includes decoding the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories.

Traffic signal understandings and representation for prediction, planning, and control

Systems and methods are provided for depth estimation from monocular images using a depth model with sparse range sensor data and uncertainty in the range sensor as inputs thereto. According to some embodiments, the methods and systems comprise receiving an image captured by an image sensor, where the image represents a scene of an environment. The method and systems also comprise deriving a point cloud representative of the scene of the environment from range sensor data, and deriving range sensor uncertainty from the range sensor data. Then a depth map can be derived for the image based on the point cloud and the range sensor uncertainty as one or more inputs into a depth model.

Depth estimation with sparse range sensor depth and uncertainty projection

Systems, methods, and other embodiments described herein relate to a multi-task model that integrates recurrent models to improve handling of multi-sweep inputs. In one embodiment, a method includes acquiring sensor data from multiple modalities. The method includes separately encoding respective segments of the sensor data according to an associated one of the different modalities to form encoded features using separate encoders of a network. The method includes accumulating, in a detector, sparse features associated with sparse sensor inputs of the multiple modalities to densify the sparse features into dense features. The method includes providing observations according to the encoded features and the sparse features using the network.

Network for multisweep 3D detection

Systems and methods are provided for training a depth model to recover scale factor for self-supervised depth estimation in monocular images. Examples include deriving a depth map for an image based on a depth model. The depth map comprises depth values for pixels of the image. A first scale for the image can be estimated based on the depth values, and depth data captured by a range sensor can be received. The depth data comprises a point cloud comprising depth measures. A second scale for the point cloud can be determined based on the depth measures and a scale factor can be determined based the second scale and the first scale. The depth model can be updated based on the scale factor, wherein the depth model generates metrically accurate depth estimates based on the scale factor.

Scale-aware self-supervised monocular depth with sparse radar supervision

A method for controlling an ego vehicle in an environment includes associating, by a velocity model, one or more objects within the environment with a respective velocity instance label. The method also includes selectively, by a recurrent network of the taillight recognition system, focusing on a selected region of the sequence of images according to a spatial attention model for a vehicle taillight recognition task. The method further includes concatenating the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The method still further planning a trajectory of the ego vehicle based on inferring, at a classifier of the taillight recognition system, an intent of each object of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label.

Vehicle taillight recognition based on a velocity estimation

A method controlling an ego vehicle in an environment includes determining, via a flow model of a parked vehicle recognition system, a flow between a first representation of the environment and a second representation of the environment. The method also includes determining, via a velocity model of the parked vehicle recognition system, a velocity of a vehicle in the environment based on the flow. The method further includes determining, via a parked vehicle classification model of the parked vehicle recognition system, the vehicle is parked based on the velocity of the vehicle and one or more of features associated with the vehicle and/or the environment. The method still further includes planning a trajectory of the ego vehicle based on determining the vehicle is parked.

Parked car classification based on a velocity estimation

A method for generating a dense light detection and ranging (LiDAR) representation by a vision system includes receiving, at a sparse depth network, one or more sparse representations of an environment. The method also includes generating a depth estimate of the environment depicted in an image captured by an image capturing sensor. The method further includes generating, via the sparse depth network, one or more sparse depth estimates based on receiving the one or more sparse representations. The method also includes fusing the depth estimate and the one or more sparse depth estimates to generate a dense depth estimate. The method further includes generating the dense LiDAR representation based on the dense depth estimate and controlling an action of the vehicle based on identifying a three-dimensional object in the dense LiDAR representation.

Shared vision system backbone

System, methods, and other embodiments described herein relate to detection of traffic lights corresponding to a driving lane from views captured by multiple cameras. In one embodiment, a method includes estimating, by a first model using images from multiple cameras, positions and state confidences of traffic lights corresponding to a driving lane of a vehicle. The method also includes aggregating, by a second model, the state confidences and a multi-view stereo composition from geometric representations associated with the positions of the traffic lights. The method also includes assigning, by the second model according to the aggregating, a relevancy score computed for a candidate traffic light of the traffic lights to the driving lane. The method also includes executing a task by the vehicle according to the relevancy score.

Systems and methods for detecting traffic lights corresponding to a driving lane

System, methods, and other embodiments described herein relate to improving the detection of traffic lights associated with a driving lane using a camera instead of map data. In one embodiment, a method includes estimating, from an image using a first model, depth and orientation information of traffic lights relative to a driving lane of a vehicle. The method also includes computing, using a second model, relevancy scores for the traffic lights according to geometric inferences between the depth and the orientation information. The method also includes assigning, using the second model, a primary relevancy score for a light of the traffic lights associated with the driving lane according to the depth and the orientation information. The method also includes executing a control task by the vehicle according to the primary relevancy score and a state confidence, computed by the first model, for the light.

Systems and methods for detecting traffic lights of driving lanes using a camera and multiple models

System, methods, and other embodiments described herein relate to accurately distinguishing a traffic light from other illuminated objects in the traffic scene and detecting states using hierarchical modeling. In one embodiment, a method includes detecting, using a machine learning (ML) model, two-dimensional (2D) coordinates of illuminated objects identified from a monocular image of a traffic scene for control adaptation by a control model. The method also includes assigning, using the ML model, computed probabilities to the illuminated objects for categories within a hierarchical ontology of environmental lights associated with the traffic scene, wherein one of the probabilities indicates existence of a traffic light instead of a brake light in the traffic scene. The method also includes executing a task by the control model for a vehicle according to the 2D coordinates and the computed probabilities.

Growing community of inventors

San Francisco, CA, United States of America

Charles Christopher Ochoa