Toyota Research Institute, Inc.

Toyota Jidosha Kabushiki Kaisha

University of Michigan

Jeffrey Michael Walls

Ryan W Wolcott

Shunsho Kaku

Steven A Parkison

Kun-Hsin Chen

Jie Li

Ryan M Eustice

Kuan-Hui Lee

Paul J Ozog

Mohammad Saad

Arash K Ushani

Fernando Nobre

Systems, methods, and other embodiments described herein relate to the detection of lanes in a driving scene through segmenting road regions using an ontology enhanced to derive semantic context. In one embodiment, a method includes segmenting an image of a driving scene, independent of maps, by lane lines and road regions defined by an ontology and a pixel subset from the image has semantics of lane information from the ontology. The method also includes computing pixel depth from the image for the lane lines and the road regions using a model. The method also includes deriving 3D context using relations between the semantics and the pixel depth, the relations infer a driving lane for a vehicle from types of the lane lines and the road regions adjacent to the driving lane. The method also includes executing a task to control the vehicle on the driving lane using the 3D context.

Systems and methods for detecting lanes using a segmented image and semantic context

Systems and methods for detecting roadway lane boundaries are disclosed herein. One embodiment receives image data of a portion of a roadway; receives historical vehicle trajectory data for the portion of the roadway; generates, from the historical vehicle trajectory data, a heatmap indicating, for a given pixel in the heatmap, an extent to which the given pixel coincides spatially with vehicle trajectories in the historical vehicle trajectory data; and projects the heatmap onto the image data to generate a composite image that is used in training a neural network to detect roadway lane boundaries, the projected heatmap acting as supervisory data. The trained neural network is deployed in a vehicle to generate and save map data including detected roadway lane boundaries for use by other vehicles or to control operation of the vehicle itself based, at least in part, on roadway lane boundaries detected by the trained neural network.

Systems and methods for detecting roadway lane boundaries

A method for an end-to-end boundary lane detection system is described. The method includes gridding a red-green-blue (RGB) image captured by a camera sensor mounted on an ego vehicle into a plurality of image patches. The method also includes generating different image patch embeddings to provide correlations between the plurality of image patches and the RGB image. The method further includes encoding the different image patch embeddings into predetermined categories, grid offsets, and instance identifications. The method also includes generating lane boundary keypoints of the RGB image based on the encoding of the different image patch embeddings.

End-to-end learned lane boundary detection based on a transformer

A method for an online mapping system includes localizing a location of an ego vehicle relative to an offline feature map. The method also includes querying surrounding features of the ego vehicle based on the offline feature map. The method further includes generating a probabilistic map regarding the surrounding features of the ego vehicle queried from the offline feature map.

Learning-based online mapping

A neural network can be configured to produce an electronic road map. The electronic road map can have information to distinguish lanes of a road. A feature in an image can be detected. The image can have been produced at a current time. The image can be of the road. The feature in the image can be determined to correspond to a feature, of a plurality of features, in a feature map. The feature map can have been produced at a prior time from one or more images. A labeled training map can be produced from the feature in the image and the plurality of features in the feature map. The labeled training map can have the information to distinguish the lanes of the road. The neural network can be trained to produce, in response to a receipt of the image and the feature map, the labeled training map.

Configuring a neural network to produce an electronic road map that has information to distinguish lanes of a road

A system for localizing a vehicle in an environment having a computing device comprising a processor and a non-transitory computer readable memory, a first map data stored in the non-transitory computer readable memory, where the first map data defines features within an environment used to localize a vehicle within the environment, and a machine-readable instruction set. The machine-readable instruction set causes the computing device to: determine a portion of the first map data having a first type of road, determine a first accuracy specification for the first type of road, wherein the first accuracy specification identifies one or more features of the plurality of features defined in the first map data used to localize a vehicle traversing the first type of road within a predefined degree of accuracy, and create a second map data for the first type of road.

Systems and methods for localizing a vehicle using an accuracy specification

Systems and methods for localization of a subject vehicle may include: a plurality of sensors to gather data regarding features in an environment of the subject vehicle; a feature map module to generate a feature map using data gathered from one or more of the plurality of sensors; a feature raster module to generate a feature raster image from the feature map; a score computation module to register a plurality of transformations of the feature raster image with a map feature raster image, to generate a cost function for the registrations of the plurality of transformations, and to determine the transformation with the lowest cost function; and the score computation module using the transformation with the lowest cost function to determine localization of the subject vehicle.

Register sets of low-level features without data association

A system for determining the rigid-body transformation between 2D image data and 3D point cloud data includes a first sensor configured to capture image data of an environment, a second sensor configured to capture point cloud data of the environment; and a computing device communicatively coupled to the first sensor and the second sensor. The computing device is configured to receive image data from the first sensor and point cloud data from the second sensor, parameterize one or more 2D lines from image data, parameterize one or more 3D lines from point cloud data, align the one or more 2D lines with the one or more 3D lines by solving a registration problem formulated as a mixed integer linear program to simultaneously solve for a projection transform vector and a data association set, and generate a data mesh comprising the image data aligned with the point cloud data.

2D to 3D line-based registration with unknown associations

A point cloud management system provides labels for each point within a point cloud map. The point cloud management system also provides a method to localize a vehicle using the labeled point cloud. The point cloud management system identifies objects within a scene using an obtained image. The point cloud management system labels the identified objects to register the identified objects against the point cloud. The registration of the objects is then used to localize the vehicle.

Locating a vehicle based on labeling point cloud data of a scene

System, methods, and other embodiments described herein relate to calibrating a light detection and ranging (LiDAR) sensor with a camera sensor. In one embodiment, a method includes controlling i) the LiDAR sensor to acquire point cloud data, and ii) the camera sensor to acquire an image. The point cloud data and the image at least partially overlap in relation to a field of view of a surrounding environment. The method includes projecting the point cloud data into the image to form a combined image. The method includes adjusting sensor parameters of the LiDAR sensor and the camera sensor according to the combined image to calibrate the LiDAR sensor and the camera sensor together.

System and method for calibrating a LIDAR and a camera together using semantic segmentation

A system and method for calibrating sensors may include one or more processors, a first sensor configured to obtain a two-dimensional image, a second sensor configured to obtain three-dimensional point cloud data, and a memory device. The memory device stores a data collection module and a calibration module. The data collection module has instructions that configure the one or more processors to obtain the two-dimensional image and the three-dimensional point cloud data. The calibration module has instructions that configure the one or more processors to determine and project a three-dimensional point cloud edge of the three-dimensional point cloud data onto the two-dimensional image edge, apply a branch-and-bound optimization algorithm to a plurality of rigid body transforms, determine a lowest cost transform of the plurality of rigid body transforms using the branch-and-bound optimization algorithm, and calibrate the first sensor with the second sensor using the lowest cost transform.

System and method for calibrating sensors of a sensor system

System, methods, and other embodiments described herein relate to predicting a presence of occluded objects from a robotic device. In one embodiment, a method includes, in response to acquiring sensor data about a surrounding environment, analyzing the sensor data to identify a perceived object in the surrounding environment by determining at least a class of the perceived object. The method includes determining a presence factor associated with the perceived object according to an observation model. The presence factor indicates a likelihood of an occluded object existing in an occluded region associated with the perceived object. The method includes controlling one or more systems of the robotic device according to the presence factor.

System and method for object detection using a probabilistic observation model

System, methods, and other embodiments described herein relate to improving mapping of a surrounding environment by a mapping vehicle. In one embodiment, a method includes identifying dynamic objects within the surrounding environment that are proximate to the mapping vehicle from sensor data of at least one sensor of the mapping vehicle. The dynamic objects are trackable objects that are moving within the surrounding environment. The method includes generating paths of the dynamic objects through the surrounding environment relative to the mapping vehicle according to separate observations of the dynamic objects embodied within the sensor data. The method includes producing a map of the surrounding environment from the paths.

System and method for mapping through inferences of observed objects

System, methods, and other embodiments described herein relate to identifying changes within a surrounding environment of a vehicle. In one embodiment, a method includes collecting, using at least one sensor of the vehicle, sensor data about the surrounding environment. The method includes analyzing the sensor data to identify features of the surrounding environment. The features are landmarks of the surrounding environment that are indicated within a map of the surrounding environment. The sensor data indicates at least measurements associated with the features as acquired by the at least one sensor. The method includes computing persistence likelihoods for the features according to a persistence model that characterizes at least prior observations of the features and relationships between the features. The persistence likelihoods indicate estimated persistences of the features to indicate whether the features likely still exist. The method includes controlling the vehicle according to the persistence likelihoods.

Growing community of inventors

Ann Arbor, MI, United States of America

Jeffrey Michael Walls