Nvidia Corporation

Deepmap Inc.

Di Zeng

Mengxi Wu

Mark Damon Wheeler

Derik Schroeter

Chen Chen

Gregory William Coombe

Jeff Adachi

Operations may comprise obtaining a plurality of light detection and ranging (LIDAR) scans of a region. The operations may also comprise identifying a plurality of LIDAR poses that correspond to the plurality of LIDAR scans. In addition, the operations may comprise identifying, as a plurality of keyframes, a plurality of images of the region that are captured during capturing of the plurality of LIDAR scans. The operations may also comprise determining, based on the plurality of LIDAR poses, a plurality of camera poses that correspond to the keyframes. Further, the operations may comprise identifying a plurality of two-dimensional (2D) keypoints in the keyframes. The operations also may comprise generating one or more three-dimensional (3D) keypoints based on the plurality of 2D keypoints and the respective camera poses of the plurality of keyframes.

Image-based keypoint generation

A vehicle computing system validates location data received from a Global Navigation Satellite System receiver with other sensor data. In one embodiment, the system calculates velocities with the location data and the other sensor data. The system generates a probabilistic model for velocity with a velocity calculated with location data and variance associated with the location data. The system determines a confidence score by applying the probabilistic model to one or more of the velocities calculated with other sensor data. In another embodiment, the system implements a machine learning model that considers features extracted from the sensor data. The system generates a feature vector for the location data and determines a confidence score for the location data by applying the machine learning model to the feature vector. Based on the confidence score, the system can validate the location data. The validated location data is useful for navigation and map updates.

Augmentation of global navigation satellite system based data

According to an aspect of an embodiment, operations may comprise receiving an approximate geographic location of a vehicle, accessing a map of a region within which the approximate geographic location of the vehicle is located, identifying a first region on the map within a first threshold distance of the approximate geographic location of the vehicle, identifying a second region on the map associated with one or more roads on the map, determining a search space on the map within which the vehicle is likely to be present, the search space representing an intersection of the first region and the second region, and determining a more accurate geographic location of the vehicle by performing a search within the search space.

Using map-based constraints for determining vehicle state

According to an aspect of an embodiment, operations may comprise accessing a set of vehicle poses of one or more vehicles; for each of the set of vehicle poses, accessing a high definition (HD) map of a geographical region surrounding the vehicle pose, with the HD map comprising a three-dimensional (3D) representation of the geographical region, determining a measure of constrainedness for the vehicle pose, with the measure of constrainedness representing a confidence for performing localization for the vehicle pose based on 3D structures surrounding the vehicle pose, and storing the measure of constrainedness for the vehicle pose; and for each of the geographical regions surrounding each of the set of vehicle poses, determining a measure of constrainedness for the geographical region based on measures of constrainedness of vehicle poses within the geographical region, and storing the measure of constrainedness for the geographical region.

Using measure of constrainedness in high definition maps for localization of vehicles

Operations of the present disclosure include obtaining a first measure of velocity of a vehicle based on a plurality of locations determined for the vehicle. The operations also include obtaining, based on IMU measurements of an inertial measurement unit (IMU) of the vehicle, a second measure of velocity of the vehicle. In addition, the operations include performing calibration of the IMU based on the first measure of velocity and the second measure of velocity.

Calibration of inertial measurement units of vehicles using localization

According to an aspect of an embodiment, operations may comprise (a) accessing a portion of a high definition (HD) map comprising a point cloud of a region through which a vehicle is driving, (b) identifying a base LIDAR from a plurality of LIDARs mounted on the vehicle, (c) for each of the LIDARs: receiving a LIDAR scan comprising a point cloud of the region, and determining a pose for the LIDAR, (d) for each LIDAR other than the base LIDAR, determining a transform for the LIDAR with respect to the base LIDAR, (e) repeating (c) to generate a plurality of samples, (f) for each of the samples, repeating (d) to determine a plurality of transforms for each LIDAR with respect to the base LIDAR, and (g) calibrating each of the LIDARs other than the base LIDAR by determining an aggregate transform for the LIDAR.

Calibration of multiple lidars mounted on a vehicle using localization based on a high definition map

According to an aspect of an embodiment, operations may comprise receiving, from a LIDAR mounted on a vehicle, a first 3D point cloud comprising points of a region around the vehicle as observed by the LIDAR. The operations may also comprise accessing an HD map comprising a second 3D point cloud comprising points of the region around the vehicle. The operations may also comprise segmenting LIDAR ground points from LIDAR non-ground points in the first 3D point cloud. The operations may also comprise segmenting map ground points from map non-ground points in the second 3D point cloud. The operations may also comprise determining a pose of the vehicle by matching the LIDAR ground points to the map ground points and by matching the LIDAR non-ground points to the map non-ground points.

Segmenting ground points from non-ground points to assist with localization of autonomous vehicles

According to an aspect of an embodiment, operations may comprise obtaining a first point cloud that includes a first point. The operations also comprises obtaining a second point cloud that is a copy of the first point cloud and that includes a second point that is a copy of the first point. The operations also comprises moving the second point cloud with respect to the first point cloud according to a first vector. The operations also comprises identifying a closest point of the first point cloud that is closest to the second point of the second point cloud. The operations also comprises determining a second vector between the closest point and the second point. The operations also comprises determining a measure of usefulness of the first point based on the first vector and the second vector. The operations also comprises indicating the measure of usefulness of the first point.

Annotating high definition map points with measure of usefulness for localization

Validation of global navigation satellite system location data with other sensor data

As an autonomous vehicle moves through a local area, pairwise alignment may be performed to calculate changes in the pose of the vehicle between different points in time. The vehicle comprises an imaging system configured to capture image frames depicting a portion of the surrounding area. Features are identified from the captured image frames, and a 3-D location is determined for each identified feature. The features of different image frames corresponding to different points in time are analyzed to determine a transformation in the pose of the vehicle during the time period between the image frames. The determined poses of the vehicle are used to generate an HD map of the local area.

Growing community of inventors

Sunnyvale, CA, United States of America

Di Zeng