Google Inc.

University of Minnesota

Esha D Nerurkar

Joel Hesch

Simon Lynen

Stergios I Roumeliotis

Sheng Zhao

Patrick Mihelich

Ryan Michael Hickman

Brian Patrick Williams

Chao X Guo

Georgios Georgiou

Laurent Tu

Ryan C DuToit

Dongfang Zheng

Kourosh Sartipi

An electronic device tracks its motion in an environment while building a three-dimensional visual representation of the environment that is used to correct drift in the tracked motion. A motion tracking module estimates poses of the electronic device based on feature descriptors corresponding to the visual appearance of spatial features of objects in the environment. A mapping module builds a three-dimensional visual representation of the environment based on a stored plurality of maps, and feature descriptors and estimated device poses received from the motion tracking module. The mapping module provides the three-dimensional visual representation of the environment to a localization module, which identifies correspondences between stored and observed feature descriptors. The localization module performs a loop closure by minimizing the discrepancies between matching feature descriptors to compute a localized pose. The localized pose corrects drift in the estimated pose generated by the motion tracking module.

System and method for concurrent odometry and mapping

An electronic device includes one or more imaging cameras. After a reset of the device or other specified event, the electronic device identifies an estimate of the device's pose based on location data such as Global Positioning System (GPS) data, cellular tower triangulation data, wireless network address location data, and the like. The one or more imaging cameras may be used to capture imagery of the local environment of the electronic device, and this imagery is used to refine the estimated pose to identify a refined pose of the electronic device. The refined pose may be used to identify additional imagery information, such as environmental features, that can be used to enhance the location based functionality of the electronic device.

Electronic device localization based on imagery

An electronic device detects and recovers from fault conditions while tracking its motion and building a map of its environment. A front-end motion tracking module detects fault conditions while tracking motion over time and provides mapping data to a back-end mapping module. The front-end motion tracking module indicates to the back-end mapping module when a fault condition has been detected and when the fault condition is no longer detected. The back-end mapping module generates maps from the mapping data and merges the maps to build a three-dimensional representation of the environment. The back-end mapping module buffers or discards any mapping data received from the front-end motion tracking module during the existence of a fault condition. The back-end mapping module merges the maps generated based on mapping data received before and after the fault condition and adds the merged maps to the three-dimensional representation of the environment.

System and method for fault detection and recovery for concurrent odometry and mapping

An electronic device merges a plurality of maps, or area description files (ADFs), by representing relationships among ADFs in an undirected graph, with vertices representing maps and edges representing transformations between maps. As the electronic device generates new ADFs, the electronic device merges each new ADF to a stored collection of ADFs by adding each new ADF as a vertex and transformations between the new ADF and the collection of ADFs as edges in the undirected graph. In this way, the map merger can use the undirected graph to more accurately represent the relations between any two maps, allowing more efficient merger of new maps to a previously stored collection of maps, and allowing for the development of more flexible and efficient algorithms for manipulating the merged maps.

System and method for merging maps

A method includes: receiving, with a computing platform, respective trajectory data and map data independently generated by each of a plurality of vision-aided inertial navigation devices (VINS devices) traversing an environment, wherein the trajectory data specifies poses along a path through the environment for the respective VINS device and the map data specifies positions of observed features within the environment as determined by an estimator executed by the respective VINS device; determining, with the computing platform and based on the respective trajectory data and map data from each of the VINS devices, estimates for relative poses within the environment by determining transformations that geometrically relate the trajectory data and the map data between one or more pairs of the VINS devices; and generating, with the computing platform and based on the transformations, a composite map specifying positions within the environment for the features observed by the VINS devices.

Resource-aware large-scale cooperative 3D mapping using multiple mobile devices

An electronic device includes one or more imaging sensors (e.g, imaging cameras) and includes one or more non-image sensors, such as an inertial measurement unit (IMU), that can provide information indicative of the pose of the electronic device. The electronic device estimates its pose based on two independent sources of pose information: pose information generated at a relatively high rate based on non-visual information generated by the non-image sensors and pose information generated at a relatively low rate based on imagery captured by the one or more imaging sensors. To achieve both a high pose-estimation rate and high degree of pose estimation accuracy, the electronic device adjusts a pose estimate based on the non-visual pose information at a high rate, and at a lower rate spatially smoothes the pose estimate based on the visual pose information.

Electronic device pose identification based on imagery and non-image sensor data

Estimation techniques for vision-aided inertial navigation are described. In one example, a vision-aided inertial navigation system (VINS) comprises an image source to produce image data for a keyframe and one or more non-keyframes along a trajectory, the one or more non-keyframes preceding the keyframe along the trajectory. The VINS comprises an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes, and a processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates of the VINS. The estimator computes the state estimates of the VINS for the keyframe by constraining the state estimates based on the IMU data and the image data for the one or more non-keyframes of the VINS without computing state estimates of the VINS for the one or more non-keyframes.

Constrained key frame localization and mapping for vision-aided inertial navigation

An electronic device reduces localization data based on feature characteristics identified from the data. Based on the feature characteristics, a quality value can be assigned to each identified feature, indicating the likelihood that the data associated with the feature will be useful in mapping a local environment of the electronic device. The localization data is reduced by removing data associated with features have a low quality value, and the reduced localization data is used to map the local environment of the device by locating features identified from the reduced localization data in a frame of reference for the electronic device.

Managing feature data for environment mapping on an electronic device

A computing system includes a network interface, a first datastore, a second datastore, and a merge module. The merge module is to receive a set of one or more area description files from a set of one or more first mobile devices. Each area description file represents a point cloud of spatial features detected by a corresponding first mobile device at an area. The computing system further includes a localization module and a query module. The localization generation module is to generate a localization area description file for the area from the set of one or more area description files and to store the localization area description file in the second datastore. The localization area description file represents a point cloud of spatial features for the area. The query module is to provide the localization area description file to a second mobile device via the network interface.

Growing community of inventors

Mountian View, CA, United States of America

Esha D Nerurkar