Zoox, Inc.

Elena Stephanie Stumm

Derek Adams

Michael Carsten Bosse

Nitesh Shroff

Brice Rebsamen

Jesse Sol Levinson

Ashutosh Gajanan Rege

Veeresh Taranalli

Patrick Blaes

Till Kroeger

Nathaniel Jon Kaiser

Qi Fu

Techniques are described for modifying a factor associated with an invalid point cloud registration (e.g., alignment) between two data sets, to improve a map generation process. A computing system receives sensor data from a vehicle traversing an environment. The computing system aligns discrete data sets, such as to generate a representation of the environment based on the discrete data sets. In some examples, the computing system can apply an optimization algorithm to the alignments, to optimize the representation of the environment. The computing system identifies an invalid (non-ideal) alignment between two data sets and modifies a factor associated with the invalid alignment, introducing a repulsive force. The computing system performs an optimization with the modified factor, which causes a location associated with at least one of the two data sets to be different from a previously measured location. The computing system generates a map based on the optimization.

Aligning data for mapping

Techniques for integrating sensor data into a scene or map based on statistical data of captured environmental data are discussed herein. The data may be stored as a multi-resolution voxel space and the techniques may comprise first applying a pre-alignment or localization technique prior to fully integrating the sensor data.

System for aligning sensor data with maps comprising covariances

Relocating and/or re-sizing map elements using an updated pose graph without introducing abnormalities to the map data may comprise determining a transformation between a source node of a first pose graph and a target node of a second pose graph and determining a modification to a map element based at least in part on the transformation. The techniques may include determining a stress on the map based at least in part on one or more modifications to map elements and determining if the stress meets or exceeds a threshold. In instances where the stress meets or exceeds a threshold, a modification may be altered, reversed, and/or indicated in a notification transmitted to a user interface.

Map annotation modification using sparse pose graph node updates

Techniques associated with generating and maintaining sparse geographic and map data. In some cases, the system may maintain a factor graph comprising a plurality of nodes. In some cases, the nodes may comprise pose data and sensor data associated with an autonomous vehicle at the geographic position represented by the node. The nodes may be linked based on shared trajectories and shared sensor data.

System for sparsely representing and storing geographic and map data

Techniques for determining a location of a vehicle in an environment using sensors and determining calibration information associated with the sensors are discussed herein. A vehicle can use map data to traverse an environment. The map data can include semantic map objects such as traffic lights, lane markings, etc. The vehicle can use a sensor, such as an image sensor, to capture sensor data. Semantic map objects can be projected into the sensor data and matched with object(s) in the sensor data. Such semantic objects can be represented as a center point and covariance data. A distance or likelihood associated with the projected semantic map object and the sensed object can be optimized to determine a location of the vehicle. Sensed objects can be determined to be the same based on matching with the semantic map object. Epipolar geometry can be used to determine if sensors are capturing consistent data.

Calibration based on semantic objects

Techniques are described for determining whether a point cloud registration (e.g., alignment) between two sets of data points is valid. Such techniques can include determining voxelized representations of the sets of data points, and comparing characteristics of spatially aligned voxels within the voxelized representations. Characteristics of voxels to be compared can include classification labels of data points associated with voxels, including whether or not voxels correspond to free space. Point cloud registrations determined to be invalid can be given a weighting to be used in a subsequent high definition (HD) map building process. Generated maps can then be deployed for use in autonomous vehicles.

LIDAR point cloud alignment validator in HD mapping

Localization based on semantic objects

Techniques are discussed for using multi-resolution maps, for example, for localizing a vehicle. Map data of an environment can be represented by discrete map tiles. In some cases, a set of map tiles can be precomputed as contributing to localizing the vehicle in the environment, and accordingly, the set of map tiles can be loaded into memory when the vehicle is at a particular location in the environment. Further, a level of detail represented by the map tiles can be based at least in part on a distance between a location associated with the vehicle and a location associated with a respective region in the environment. The level of detail can also be based on a speed of the vehicle in the environment. The vehicle can determine its location in the environment based on the map tiles and/or the vehicle can generate a trajectory based on the map tiles.

Multi-resolution maps for localization

Techniques for decimating portions of a map of an environment are discussed herein. The environment can be represented by a three-dimensional (3D) map including a plurality of polygons and semantic information associated with the polygons. In some cases, decimation operations may be based on semantic information associated with the environment. Differing decimation operations and/or levels may be applied to polygons of different semantic classifications or differing contribution levels. Boundaries between regions having different semantic information can be preserved. Meshes can be decimated using different decimation operators or decimation levels and an accuracy of localizing can be compared using the various decimated meshes. An optimal mesh can be selected and sent to vehicles for localizing the vehicles in the environment.

Mesh decimation based on semantic information

A vehicle may include a localization and/or mapping component to understand what surrounds the autonomous vehicle and where it is in relation to the surroundings. The localization and/or mapping component may receive sensor data from sensor(s) of the vehicle and generate a map and/or position and/or orientation from the sensor data according to parameters that configure the way the localization and/or mapping component generates the map and/or position/orientation. A computing device may dynamically adjust these parameters, thereby changing the way the map and/or position/orientation are generated. This adjustment may be based on a condition detected from the sensor data and may increase a clarity (e.g., degree of distinctness/clarity) of the generated map and/or position/orientation.

Condition dependent parameters for large-scale localization and/or mapping

Loading multi-resolution maps for localization

Mesh decimation techniques and validation

Techniques for generating maps without shadows are discussed herein. A plurality of images can be captured by a vehicle traversing an environment representing various perspectives and/or lighting conditions in the environment. A shadow within an image can be identified by a machine learning algorithm trained to detect shadows in images and/or by projecting the image onto a three-dimensional (3D) map of the environment and identifying candidate shadow regions based on the geometry of the 3D map and the location of the light source. Shadows can be removed or minimized by utilizing blending or duplicating techniques. Color information and reflectance information can be added to the 3D map to generate a textured 3D map. A textured 3D map without shadows can be used to simulate the environment under different lighting conditions.

Growing community of inventors

San Francisco, CA, United States of America

Elena Stephanie Stumm