Toyota Research Institute, Inc.

Toyota Jidosha Kabushiki Kaisha

University of Michigan

Steven A Parkison

Jeffrey Michael Walls

Kun-Hsin Chen

Shunsho Kaku

Jie Li

Ryan W Wolcott

Ryan M Eustice

Kuan-Hui Lee

Mohammad Saad

Systems, methods, and other embodiments described herein relate to the detection of lanes in a driving scene through segmenting road regions using an ontology enhanced to derive semantic context. In one embodiment, a method includes segmenting an image of a driving scene, independent of maps, by lane lines and road regions defined by an ontology and a pixel subset from the image has semantics of lane information from the ontology. The method also includes computing pixel depth from the image for the lane lines and the road regions using a model. The method also includes deriving 3D context using relations between the semantics and the pixel depth, the relations infer a driving lane for a vehicle from types of the lane lines and the road regions adjacent to the driving lane. The method also includes executing a task to control the vehicle on the driving lane using the 3D context.

Systems and methods for detecting lanes using a segmented image and semantic context

Systems and methods for detecting roadway lane boundaries are disclosed herein. One embodiment receives image data of a portion of a roadway; receives historical vehicle trajectory data for the portion of the roadway; generates, from the historical vehicle trajectory data, a heatmap indicating, for a given pixel in the heatmap, an extent to which the given pixel coincides spatially with vehicle trajectories in the historical vehicle trajectory data; and projects the heatmap onto the image data to generate a composite image that is used in training a neural network to detect roadway lane boundaries, the projected heatmap acting as supervisory data. The trained neural network is deployed in a vehicle to generate and save map data including detected roadway lane boundaries for use by other vehicles or to control operation of the vehicle itself based, at least in part, on roadway lane boundaries detected by the trained neural network.

Systems and methods for detecting roadway lane boundaries

A method for an end-to-end boundary lane detection system is described. The method includes gridding a red-green-blue (RGB) image captured by a camera sensor mounted on an ego vehicle into a plurality of image patches. The method also includes generating different image patch embeddings to provide correlations between the plurality of image patches and the RGB image. The method further includes encoding the different image patch embeddings into predetermined categories, grid offsets, and instance identifications. The method also includes generating lane boundary keypoints of the RGB image based on the encoding of the different image patch embeddings.

End-to-end learned lane boundary detection based on a transformer

A system for determining the rigid-body transformation between 2D image data and 3D point cloud data includes a first sensor configured to capture image data of an environment, a second sensor configured to capture point cloud data of the environment; and a computing device communicatively coupled to the first sensor and the second sensor. The computing device is configured to receive image data from the first sensor and point cloud data from the second sensor, parameterize one or more 2D lines from image data, parameterize one or more 3D lines from point cloud data, align the one or more 2D lines with the one or more 3D lines by solving a registration problem formulated as a mixed integer linear program to simultaneously solve for a projection transform vector and a data association set, and generate a data mesh comprising the image data aligned with the point cloud data.

2D to 3D line-based registration with unknown associations

A system and method for calibrating sensors may include one or more processors, a first sensor configured to obtain a two-dimensional image, a second sensor configured to obtain three-dimensional point cloud data, and a memory device. The memory device stores a data collection module and a calibration module. The data collection module has instructions that configure the one or more processors to obtain the two-dimensional image and the three-dimensional point cloud data. The calibration module has instructions that configure the one or more processors to determine and project a three-dimensional point cloud edge of the three-dimensional point cloud data onto the two-dimensional image edge, apply a branch-and-bound optimization algorithm to a plurality of rigid body transforms, determine a lowest cost transform of the plurality of rigid body transforms using the branch-and-bound optimization algorithm, and calibrate the first sensor with the second sensor using the lowest cost transform.

Growing community of inventors

Ann Arbor, MI, United States of America

Steven A Parkison