Nvidia Corporation

Deepmap Inc.

Microsoft Technology Licensing, LLC

Lin Yang

Mark Damon Wheeler

Yu Zhang

Derek Thomas Miller

Ronghua Zhang

Kevin Chang

Ziqiang Huang

Xiaqing Wu

Lenord Melvix Joseph Stephen Max

Ming Cui

Qian Gong

Dongzhen Piao

Zhengyu Zhang

A system accesses a three-dimensional map of a geographic region and generates a two-dimensional projection of the road based on the three-dimensional map. The two-dimensional projection comprises a plurality of points along the road and each point is assigned a score measuring a navigability of the point. Based on the navigability score of each point and history of vehicle positions on the road, the system identifies a plurality of navigable points on the two-dimensional projection of the road. Based on the plurality of navigable points, the system determines a navigable surface corresponding to a physical area over which a vehicle may safely navigate and navigable surface boundaries surrounding that area. The navigable surface area and boundaries on the two-dimensional projection are converted into a three-dimensional representation, which the system uses to generate an updated three-dimensional map of the geographic region.

Navigable boundary generation for autonomous vehicles

According to an aspect of an embodiment, a method may include obtaining multiple sets of camera images and light detection and ranging (LIDAR) point clouds along a track within a geographic sector of a map. The method may include applying a learning model to the camera images to characterize objects within the camera images within classes of objects to generate segmented images. The method may additionally include mapping the sets of camera images and the LIDAR point clouds to three dimensional points of the geographic sector of the map. The method may also include projecting the three dimensional points onto the segmented images to obtain corresponding classes for the three dimensional points of the geographic sector of the map.

Annotating high definition map data with semantic labels

According to an aspect of an embodiment, operations may comprise receiving sensor data from one or more vehicles, determining, by combining the received sensor data, a high definition map comprising a point cloud, and labeling one or more objects in the point cloud. The operations may also comprise generating training data by receiving a new image captured by one of the vehicles, receiving a pose of the vehicle when the new image was captured, determining an object having a label in the point cloud that is observable from the pose of the vehicle, determining a position of the object in the new image, and labeling the new image by assigning the label of the object to the new image, the labeled new image comprising the training data. The operations may also comprise training a deep learning model using the training data.

Generating training data for deep learning models for building high definition maps

Embodiments relate to methods for efficiently encoding sensor data captured by an autonomous vehicle and building a high definition map using the encoded sensor data. The sensor data can be LiDAR data which is expressed as multiple image representations. Image representations that include important LiDAR data undergo a lossless compression while image representations that include LiDAR data that is more error-tolerant undergo a lossy compression. Therefore, the compressed sensor data can be transmitted to an online system for building a high definition map. When building a high definition map, entities, such as road signs and road lines, are constructed such that when encoded and compressed, the high definition map consumes less storage space. The positions of entities are expressed in relation to a reference centerline in the high definition map. Therefore, each position of an entity can be expressed in fewer numerical digits in comparison to conventional methods.

Encoding LiDAR scanned data for generating high definition maps for autonomous vehicles

According to an aspect of an embodiment, operations may comprise receiving a LIDAR scan of a scene from a LIDAR of a vehicle with the scene comprising a board, detecting the board in the LIDAR scan, fitting a plane through LIDAR coordinates corresponding to the detected board, projecting the plane from the LIDAR coordinates to a first set of camera coordinates, detecting the board in a camera image from a camera of the vehicle at a second set of camera coordinates, and calibrating the LIDAR of the vehicle and the camera of the vehicle by determining a transform between the first set of camera coordinates and the second set of camera coordinates.

LIDAR-to-camera transformation during sensor calibration for autonomous vehicles

Camera based localization performed to determine a current pose of an autonomous vehicle without the aid of depth sensors such as LiDAR. The vehicle comprises an imaging system configured to capture image frames depicting portions of the surrounding area. Based on an initial pose of the vehicle, edgels corresponding to three-dimensional locations are loaded and mapped to corresponding edge pixels of the captured image frame. A pose of the vehicle is optimized based upon the determined correspondences by identifying a transformation that minimizes a distance between the edgels and their corresponding edge pixels. The determined transformation can be applied to the initial pose to determine an updated pose of the vehicle.

Camera based localization for autonomous vehicles

A system calibrates one or more sensors mounted to an autonomous vehicle. From the one or more sensors, the system identifies a primary sensor and a secondary sensor. The system determines a reference angle for the primary sensor, and based on that reference angle for the primary sensor, a scan-start time representing a start of a scan and a scan-end time representing an end of a scan. The system receives, from the primary sensor, a primary set of scan data recorded from the scan-start time to the scan-end time. The system receives, from the secondary sensor, a secondary set of sensor data recorded from the scan-start time to the scan-end time. The system calibrates the primary and secondary sensors by determining a relative transform for transforming points between the first set of scan data and the second set of scan data.

Calibrating sensors mounted on an autonomous vehicle

According to an aspect of an embodiment, operations may comprise receiving a point cloud representing a region. The operations may also comprise identifying a cluster of points in the point cloud having a higher intensity than points outside the cluster of points. The operations may also comprise determining a bounding box around the cluster of points. The operations may also comprise identifying a traffic sign within the bounding box. The operations may also comprise projecting the bounding box to coordinates of an image of the region captured by a camera. The operations may also comprise employing a deep learning model to classify a traffic sign type of the traffic sign in a portion of the image within the projected bounding box. The operations may also comprise storing information regarding the traffic sign and the traffic sign type in a high definition (HD) map of the region.

LIDAR-based detection of traffic signs for navigation of autonomous vehicles

According to an aspect of an embodiment, operations may comprise capturing, at a vehicle as the vehicle travels, LIDAR scans and camera images. The operations may further comprise selecting, at the vehicle as the vehicle travels, a subset of the LIDAR scans and the camera images that are determined to be useful for calibration. The operations may further comprise computing, at the vehicle as the vehicle travels, LIDAR-to-camera transformations for the subset of the LIDAR scans and the camera images using an optimization algorithm. The operations may further comprise calibrating, at the vehicle as the vehicle travels, one or more sensors of the vehicle based on the LIDAR-to-camera transformations.

Online sensor calibration for autonomous vehicles

According to an aspect of an embodiment, operations may comprise determining a target position and orientation for a calibration board with respect to a camera of a vehicle, detecting a first position and orientation of the calibration board with respect to the camera of the vehicle, determining instructions for moving the calibration board from the first position and orientation to the target position and orientation, transmitting the instructions to a device, detecting a second position and orientation of the calibration board, determining whether the second position and orientation is within a threshold of matching the target position and orientation, and, in response to determining that the second position and orientation is within the threshold of matching the target position and orientation, capturing one or more calibration camera images using the camera and calibrating one or more sensors of the vehicle using the one or more calibration camera images.

Directing board repositioning during sensor calibration for autonomous vehicles

Encoding lidar scanned data for generating high definition maps for autonomous vehicles

A method and system for synchronizing a lidar and a camera on an autonomous vehicle. The system instructs the camera to detect light columns transmitted by the lidar. The system iterates through various start times for the camera. The system instructs the lidar to emit a plurality of light columns at a lidar frequency. The system instructs the camera to capture images at a camera frequency starting at each start time. The system analyzes the image data received from the cameras to identify light columns captured in the images. The system calculates an alignment score for each of the many start times based on the identified light columns. The start time with the optimal alignment score is selected and used to synchronize the lidar and the camera. With lidar data detected by the synchronized lidar and image data captured by the synchronized camera, the system may navigate the autonomous vehicle.

Growing community of inventors

San Carlos, CA, United States of America

Lin Yang