Denso International America, Inc.

Denso Corporation

Magna Electronics Inc.

Carnegie Mellon University

Roboticvisiontech LLC

Denso Corpora Tion

Vision Guided Robotics LLC

Shawn T Hunt

Yuesheng Lu

Joel S Gibson

Prasanna Sivakumar

Travis G Johnson

Joseph Lull

Kris Kitani

Matthew O'Toole

Xinshuo Weng

Dave Coker

Nelson D'Agostini

Matthew Patrick O'toole

Yunze Man

Yu-Jhe Li

A method for calibrating a radar system comprising one or more radar sensors includes defining, for a given calibration iteration from among a plurality of calibration iterations, one or more operational characteristics of the one or more radar sensors and generating a range-azimuth map based on radar data generated by the one or more radar sensors. The method includes identifying a reference set of pixels from among the plurality of pixels, where the reference set of pixels is associated with a reference object, determining whether a set of intensity values associated with the reference set of pixels satisfies an intensity condition, and adjusting the one or more operational characteristics of the one or more radar sensors in response to the set of intensity values not satisfying the intensity condition.

Systems and methods for calibrating a radar system

A method includes generating a radar-based intensity map and a lidar-based intensity map and performing one or more augmentation routines on the radar-based intensity map and the lidar-based intensity map to generate a radar input and a lidar input. The method includes generating a plurality of teacher-based bounding boxes and a plurality of student-based bounding boxes based on the radar input and the lidar input. The method includes determining a loss value of the plurality of student-based bounding boxes based on the plurality of teacher-based bounding boxes and a plurality of ground truth bounding boxes, updating one or more weights of the student neural network based on the loss value, and updating one or more weights of the teacher neural network based on a moving average associated with the one or more weights of the student neural network.

System and method for training a neural network to perform object detection using lidar sensors and radar sensors

A method includes generating a plurality of lidar inputs based on the lidar data, where each lidar input from among the plurality of lidar inputs comprises an image-based portion and a geometric-based portion, and where each lidar input from among the plurality of lidar inputs defines a position coordinate of the one or more objects. The method includes performing, for each lidar input from among the plurality of lidar inputs, a convolutional neural network (CNN) routine based on the image-based portion to generate one or more image-based outputs and assigning the plurality of lidar inputs to a plurality of echo groups based on the geometric-based portion. The method includes concatenating the one or more image-based outputs and the plurality of echo groups to generate a plurality of fused outputs and identifying the one or more objects based on the plurality of fused outputs.

Systems and methods for detecting objects based on lidar data

A method of calibrating a camera sensor and a SPAD LiDAR includes extracting identified features in each of a selected camera image and an ambient-intensity (A-I) image, generating a set of keypoints based on the identified features extracted for each of the images to provide a set of 2D camera keypoint locations and a set of 2D A-I keypoint locations, determining matched keypoints based on the set of 2D A-I keypoint locations and the set of 2D camera keypoint locations to provide a set of 2D A-I matched pixel locations and a set of 2D camera matched pixel locations, interpolating a 3D point cloud data with the set of 2D A-I matched pixel locations to obtain a set of 3D LiDAR matched pixel locations, and determining and storing extrinsic parameters to transform the set of 3D LiDAR matched pixel locations with the set of 2D camera matched pixel locations.

Method and system for automatic extrinsic calibration of SPAD LiDAR and camera pairs

A method of generating virtual sensor data of a virtual single-photon avalanche diode (SPAD) lidar sensor includes generating a two-dimensional (2D) lidar array having a plurality of cells. The method further includes interpolating image data from a virtual camera with the 2D lidar array to define auxiliary image data, generating a virtual ambient image based on a red-channel (R-channel) data of the auxiliary image data, identifying a plurality of virtual echoes of the virtual SPAD lidar sensor based on the R-channel data and a defined photon threshold, defining a virtual point cloud indicative of virtual photon measurements of the virtual SPAD lidar sensor based on the plurality of virtual echoes, and outputting data indicative of the virtual ambient image, the virtual photon measurements, the virtual point cloud, or a combination thereof, as the virtual sensor data of the virtual SPAD lidar sensor.

Methods and system for generating virtual sensor data of a virtual single-photon avalanche diode (SPAD) lidar sensor of a virtual vehicle simulator

System, methods, and other embodiments described herein relate to improving the synchronizing of systems in a vehicle without relying on external controls or hardware. In one embodiment, a method includes detecting a stop of a vehicle according to data acquired from two or more sensors. The method also includes identifying motion from the stop according to the data and corresponding timestamps from the two or more sensors, wherein the data and the corresponding timestamps are internal to the two or more sensors. The method also includes synchronizing the two or more sensors according to the corresponding timestamps. The method also includes controlling the motion of the vehicle using the two or more sensors.

Systems and methods to synchronize systems of a vehicle

A method for backing up a trailer hitched to a vehicle includes mounting a rear backup camera at a vehicle and hitching a trailer at the hitch of the vehicle. Responsive to processing captured image data by a processor of the control, at least a portion of the trailer tongue of the trailer is detected using an edge detection algorithm. Responsive to processing at the control of image data captured by the rear backup camera during a backing up maneuver of the trailer hitched to the vehicle, trailer angle of the trailer relative to a longitudinal axis of the vehicle is determined. The control, based at least in part on trailer angle of the trailer, controls a steering system of the vehicle during the backing up maneuver of the trailer hitched to the vehicle to back up the trailer hitched to the vehicle along a rearward trajectory.

Trailering assist system with trailer angle detection

A camera control system is provided for controlling operation of a camera mounted on a vehicle. The system includes a processor and a memory communicably coupled to the processor. The memory stores a camera control module configured to compare a recognition confidence level associated with a feature to a predetermined threshold and, responsive to the recognition confidence level being below the predetermined threshold and using location information associated with the feature, control operation of the camera to capture at least one image of the feature during movement of the vehicle.

System for controlling a camera supporting human review of sensor information

A system for generating synthetic training data may include one or processors and a memory in communication with the one or more processors and having a receiving module, a duplication module, and an insertion module. The modules have instructions that when executed by the one or more processors cause the one or more processors to receive original training data being in the form of a three-dimensional point cloud and having one or more original objects formed by at least a portion of the three-dimensional point cloud and annotated with original annotation data, duplicate one of the one or more original objects to generate a synthetic object, and insert the synthetic object within the original training data to generate the synthetic training data.

System and method for generating synthetic training data

A system for classifying an object may include one or more processors, a sensor and a memory device. The memory device may include a data collection module, a starburst module, and an object classifying module. The modules have instructions that when executed by the one or more processors cause the one or more processors to obtain three dimensional point cloud data from the sensor, identify at least one cluster of points representing the object within the three dimensional point cloud data, identify a center point of the at least one cluster of points, project a plurality of rays from the center point to points of the at least one cluster of points to generate a shape, compare the shape to a plurality of candidate shapes, and classify the object when the shape matches at least one of the plurality of candidate shapes.

System and method for classifying an object using a starburst algorithm

A system for utilizing object tracking algorithms for tracking objects external to a vehicle includes a processor and a memory in communication with the processor. The object tracking algorithms include a lower computationally demanding object tracking algorithm and a higher computationally demanding object tracking algorithm. The memory has one or more modules that have instructions that cause the processor to determine if the vehicle is located in an urban setting or a non-urban setting and utilize the lower computationally demanding object tracking algorithm by an object tracking system when the vehicle is located in the non-urban setting. If it is determined that the vehicle is in an urban setting, the instructions cause the processor to determine available computational resources of the object tracking system and utilizing either the lower computationally demanding object tracking algorithm or the higher computationally demanding object tracking algorithm based on the available computational resources.

Object tracking algorithm selection system and method

A system includes a processor and a memory in communication with the processor having one or more modules. The modules include instructions that cause the processor to receive first sensor information from a first sensor and second sensor information of a shared field of view. The instructions cause the processor to detect one or more objects within the shared field of view using the first sensor information and the second sensor information. When one or more discrepancies are identified between the one or more objects detected using the first sensor information and the one or more objects detected using the second sensor information are detected, the instructions that cause the processor to determine that the first sensor or the second sensor needs cleaning.

System and method for determining when a sensor is polluted

A system for improving inspection systems is provided. The system generally includes a camera in communication with a processor. The camera is configured to capture an image for inspection of an object A to be inspected. The camera transfers the image to the processor for comparison with a reference image. If the comparison is not within a predetermined range, then the camera settings are adjusted and a new image is taken until the current image matches the reference image. Alternatively, or in addition to, the system is capable of learning based on a plurality of data and creating a model image for comparison prior to capturing the first image.

Inspection system

A method for determining a trailer angle for a trailering assist system of a vehicle includes providing a rearward facing camera at a rearward portion of a vehicle and a control that processes image data captured by the camera. Responsive to processing by the processor of captured image data, a location of a portion of the trailer relative to the vehicle and a trailer angle of the trailer relative to a longitudinal axis of the vehicle are determined. During a reversing maneuver, a rearward trajectory of the trailer is determined and a graphic overlay may be generated to overlay displayed images to indicate the determined rearward trajectory. During forward travel, a degree of swing of the trailer may be determined and a threshold level of swing may be adjusted based at least in part on a speed of the vehicle and trailer and/or a load of the trailer.

A system for calibrating at least one camera and a light detection and ranging ('LiDAR') sensor includes one or more processors and a memory in communication with the one or more processors that stores an initial calibration module and a user calibration module. The initial calibration module includes instructions that cause the one or more processors to obtain a camera image from the at least one camera, determine when the camera image includes at least one object having at least one edge, obtain a three-dimensional point cloud image from the LiDAR sensor that includes the at least one object having at least one edge and generate a combined image, that includes at least portions of the camera image and at least portions of the three-dimensional point cloud image. The user calibration module includes instructions that cause the one or more processors to display the combined image on a display, receive at least one input from a user interface, and adjust a calibration parameter of at least one of the LiDAR sensor and the at least one camera in response to the one or more inputs from the user interface.

Growing community of inventors

Pinckney, MI, United States of America

Shawn T Hunt