Toyota Research Institute, Inc.

Yusuke Kanzawa

Kuan-Hui Lee

Michael James Delp

Michael R James

Junya Ueno

Matthew Derry

Mark Edward Tjersland

In one embodiment, a system for removing LiDAR points is provided. An image is received from a camera and a plurality of points is received from a LiDAR sensor. The points are placed on the image based on coordinates associated with each point. The image is divided into a plurality of cells by placing a grid over the image. For each cell, a threshold is calculated based on the minimum distance between the points in the cell and the camera. The threshold may control how many points are allowed to remain in each cell. After calculating the thresholds, points are removed from each cell until the number of points in each cell do not exceed the threshold for the cell. The image and/or the reduced plurality of points are then used to provide one or more vehicle functions.

Systems and methods for reducing LiDAR points

In one embodiment, a system for correcting parallax error is provided. An image is received from a camera and a plurality of points is received from a LiDAR sensor. The points are placed on the image based on coordinates associated with each point. The image is divided into a plurality of cells by placing a grid over the image. For each cell, a minimum distance between the points in the cell and the camera is determined. For each cell, a margin is calculated based on the determined minimum distance. For each cell, points that have a distance from the camera that is greater than the minimum distance plus the margin are removed or deleted. The image and/or the remaining points are then used to provide one or more vehicle functions.

Systems and methods for correcting parallax

In one embodiment, a system for determining the direction of objects in an image is provided. An image is received from a camera associated with a vehicle. The image is processed to identify an object, such as another vehicle. To determine the direction the object is traveling, the image is provided as input to a machine learning algorithm that outputs a direction of the detected object based at least in part on a training dataset including a plurality of images annotated to include respective direction information determined based at least in part on a plurality of 3D models set to aligned orientations that are significantly aligned with orientations of objects captured in the plurality of images. The objects and their associated directions can be provided to one or more autonomous vehicle functions, such as hazard detection or navigation.

Systems and methods for determining the direction of an object in an image

In one embodiment, a system for associating LiDAR points with objects is provided. Objects in an image are identified and bounding boxes are placed on each object in the image. LiDAR points are received and are projected into the image based on a reference frame of the image. All pixels of the image that are not inside a bounding box are changed to a color such as black. All points that are inside a bounding box are changed to a color that is unique to the particular bounding box. To determine which LiDAR points are associated with the objects in the image, for each LiDAR point a color of the pixel having the same coordinates in the image as the LiDAR point is determined. If the determined color is black than the LiDAR point is not associated with any object. Else, the LiDAR point is located in the bounding box that uses the determined color, and the LiDAR point can be labeled as being located with the object corresponding to the bounding box.

Systems and methods for associating LiDAR points with objects

Systems and methods described herein relate to registering three-dimensional (3D) data with two-dimensional (2D) image data. One embodiment receives 3D data from one or more sensors and 2D image data from one or more cameras; identifies a 3D segment in the 3D data and associates it with an object; classifies pixels in the 2D image data; determines a speed and a heading for the object; and registers the 3D segment with a portion of the classified pixels by either (1) shifting the 3D segment to a position that, based on the associated object's speed and heading, corresponds to a 2D image data capture time and projecting the time-shifted 3D segment onto 2D image space; or (2) projecting the 3D segment onto 2D image space and shifting the projected 3D segment to a position that, based on the associated object's speed and heading, corresponds to a 2D image data capture time.

Systems and methods for registering 3D data with 2D image data

Systems and methods described herein relate to registering three-dimensional (3D) data with two-dimensional (2D) image data. One embodiment receives 3D data from one or more sensors and 2D image data from one or more cameras; identifies a 3D segment in the 3D data and associates it with an object; identifies 2D boundary information for the object; determines a speed and a heading for the object; and registers the 3D segment with the 2D boundary information by adjusting the relative positions of the 3D segment and the 2D boundary information based on the speed and heading of the object and matching, in 3D space, the 3D segment with projected 2D boundary information.

This disclosure describes various embodiments for determining a second stop position for an intersection turn by an autonomous vehicle. In an embodiment an autonomous control module is described. The autonomous control module may comprise a memory and a processor coupled to the memory. The processor may be configured to determine a first stop location at an intersection; cause a vehicle to stop at the first stop location; determine a turn path through the intersection; determine a second stop location along the turn path; cause the vehicle to advance to the second stop location; cause the vehicle to stop at the second stop location; and cause the vehicle to continue along the turn path.

Second stop position for intersection turn

System, methods, and other embodiments described herein relate to associating disparate tracks from multiple sensor inputs for observed objects. In one embodiment, a method includes, in response to receiving a first input from a first sensor and a second input from a second sensor, generating the disparate tracks including first sensor tracks and second sensor tracks for the observed objects that correspond to the first input and the second input. The method includes identifying correlations between the first sensor tracks and the second sensor tracks by computing association likelihoods between the first tracks and the second tracks within a permutation matrix according to an objective cost function. The method includes controlling a vehicle according to the correlations.

Systems and methods for multi-sensor fusion using permutation matrix track association

System, methods, and other embodiments described herein relate to locating an object within an image and projecting the object into a map. In one embodiment, a method includes, responsive to identifying a nearby vehicle in an image of an external environment surrounding a scanning vehicle, determining a relative location of the nearby vehicle in relation to the scanning vehicle using the image. The method includes projecting the nearby vehicle into a map according to at least the relative location determined from the image. The method includes controlling the scanning vehicle according to the map.

Growing community of inventors

Ann Arbor, MI, United States of America

Yusuke Kanzawa