Google Inc.

Honeywell International Inc.

Magic Leap, Inc.

Dennis William Strelow

Alan Bruce Touchberry

Jay Yagnik

Sergey Ioffe

Lei Huang

Luc Vincent

Ashwin Swaminathan

Andrea Lynn Frome

Jean-Yves Bouguet

Etienne Gregoire Grossmann

Divya Sharma

Evan Gregory Levine

Yu-Tseh Chi

Lance Williams

Adam Harmat

Saad J Bedros

Craig Lewin Robinson

Vassilios Morellas

Kartik B Ariyur

Sally A Goldman

Luca Bertelli

Samuel Felix De Sousa, Jr

Etienne Gregoire Grossman

Techniques are described for calibrating a device having a first sensor and a second sensor. Techniques include capturing sensor data using the first sensor and the second sensor. The device maintains a calibration profile including a translation parameter and a rotation parameter to model a spatial relationship between the first sensor and the second sensor. Techniques include determining a calibration level associated with the calibration profile at a first time. Techniques include determining, based on the calibration level, to perform a calibration process. Techniques include performing the calibration process at the first time by generating one or both of a calibrated translation parameter and a calibrated rotation parameter and replacing one or both of the translation parameter and the rotation parameter with one or both of the calibrated translation parameter and the calibrated rotation parameter.

Calibration of augmented reality device

A method for calibrating a device having a first sensor and a second sensor. The method includes capturing sensor data using the first sensor and the second sensor. The device maintains a calibration profile including a translation parameter and a rotation parameter to model a spatial relationship between the first sensor and the second sensor. The method also includes determining a calibration level associated with the calibration profile at a first time. The method further includes determining, based on the calibration level, to perform a calibration process. The method further includes performing the calibration process at the first time by generating one or both of a calibrated translation parameter and a calibrated rotation parameter and replacing one or both of the translation parameter and the rotation parameter with one or both of the calibrated translation parameter and the calibrated rotation parameter.

Method for calibrating an augmented reality device

One of the described methods includes receiving a plurality of images from a camera, the plurality of images comprising a sequence; identifying one or more two-dimensional features in each of a plurality of images in the received sequence of images; associating a three-dimensional point with each of the identified one or more two-dimensional features; tracking each of the one or more two-dimensional features through successive images in the plurality of images; and iteratively minimizing a two-dimensional image error between the tracked each of the one or more two-dimensional features and an image reprojection with respect to the three-dimensional point corresponding to the one or more two-dimensional features and a three-dimensional position of the camera corresponding to one or more of the plurality of images.

General and nested Wiberg minimization

Systems and methods for aligning panoramic imagery of a geographic area captured from a perspective at or near ground level and aerial imagery captured from an oblique perspective are provided. More particularly, a facade plane can be detected in a panoramic image based at least in part on geometric data associated with the image. The panoramic image can also have an associated image pose. An aerial image depicting the same facade can then be identified. The aerial image can be associated with an image pose and geometric data of the depicted imagery. The panoramic image can be warped into an image having a perspective associated with the aerial image. One or more feature matches between the warped image and the aerial image can be identified using a feature matching technique. The matched features can be used to align the panoramic image with the aerial image.

Aligning panoramic imagery and aerial imagery

A method for determining a salient region of an image is disclosed. For a plurality of different saliency cue functions, a single saliency value is calculated for each pixel in a plurality of adjacent pixels in an image using the saliency cue function, wherein one of the saliency cue functions is based on whether the pixel is in a region of the image whose colors contrast with the region's background and another of the saliency cue functions is based on a foreground and background color models of the image. A classifier is used to calculate a combined single saliency value for each pixel based on the single saliency values for the pixel. The salient region of the pixels is determined with a subwindow search based on the combined single saliency values.

Foreground object detection from multiple images

Wiberg minimization operates on a system with two sets of variables described by a linear function and in which some data or observations are missing. The disclosure generalizes Wiberg minimization, solving for a function that is nonlinear in both sets of variables, U and V, iteratively. In one embodiment, defining a first function &#x192;(U, V) that may be defined that may be nonlinear in both a first set of variables U and a second set of variables V. A first function &#x192;(U, V) may be transformed into &#x192;(U, V(U)). First assumed values of the first set of variables U may be assigned. The second set of variables V may be iteratively estimated based upon the transformed first function &#x192;(U, V(U)) and the assumed values of the first set of variables U such that &#x192;(U, V(U)) may be minimized with respect to V. New estimates of the first set of variables U may be iteratively computed.

A method and system of identity masking to obscure identities corresponding to face regions in an image is disclosed. A face detector is applied to detect a set of possible face regions in the image. Then an identity masker is used to process the detected face regions by identity masking techniques in order to obscure identities corresponding to the regions. For example, a detected face region can be blurred as if it is in motion by a motion blur algorithm, such that the blurred region can not be recognized as the original identity. Or the detected face region can be replaced by a substitute facial image by a face replacement algorithm to obscure the corresponding identity.

Automatic face detection and identity masking in images, and applications thereof

A method and system for navigation of one or more unmanned aerial vehicles in an urban environment is provided. The method comprises flying at least one Global Positioning System (GPS)-aided unmanned aerial vehicle at a first altitude over an urban environment, and flying at least one GPS-denied unmanned aerial vehicle at a second altitude over the urban environment that is lower than the first altitude. The unmanned aerial vehicles are in operative communication with each other so that images can be transmitted therebetween. A first set of images from the GPS-aided unmanned aerial vehicle is captured, and a second set of images from the GPS-denied unmanned aerial vehicle is also captured. Image features from the second set of images are then matched with corresponding image features from the first set of images. A current position of the GPS-denied unmanned aerial vehicle is calculated based on the matched image features from the first and second sets of images.

Method and system for GPS-denied navigation of unmanned aerial vehicles

A system for autonomous object tracking with static camera arrangements. Each camera arrangement may minimally have a pan-tilt-zoom camera and a range or depth sensor. Imaging may provide coordinates and depth information of a tracked object. Measurements of an image centroid position and width may be obtained with processing. Maintaining an image at the center of a camera screen may be attained at a pixel width of the image. Estimation and prediction of object size and position may be processed for providing pan, tilt and zoom rates for the camera. Pan, tilt and zoom latency may be accounted for in the system. There may be a number of camera arrangements where tracking of the object may be handed off by one camera arrangement to another.

System and method for autonomous object tracking

A method and system are provided for autonomous vehicle navigation. In the method and system, one or more Global Positioning System (GPS) sensors, one or more inertial sensors, and one or more image sensors are provided on an autonomous vehicle. During operation of the autonomous vehicle, one or more GPS measurements, one or more inertial measurements, and one or more image measurements are obtained from the respective GPS sensors, inertial sensors, and image sensors. Each of the GPS measurements, inertial measurements, and image measurements are integrated together to estimate a position of the autonomous vehicle over time.

Method and system for autonomous vehicle navigation

A method and system for navigation of an unmanned aerial vehicle (UAV) in an urban environment are provided. The method comprises capturing a first set of Global Positioning System (GPS)-tagged images in an initial fly-over of the urban environment at a first altitude, with each of the GPS-tagged images being related to respective GPS-aided positions. The captured GPS-tagged images are stitched together into an image mosaic using the GPS-aided positions. A second set of images is captured in a subsequent fly-over of the urban environment at a second altitude that is lower than the first altitude. Image features from the second set of images are matched with image features from the image mosaic during the subsequent fly-over. A current position of the UAV relative to the GPS-aided positions is calculated based on the matched image features from the second set of images and the image mosaic.

Method and system for navigation of an ummanned aerial vehicle in an urban environment

The dataset describing an entity in a first modality and of a first, high resolution is used to enhance the resolution of a dataset describing the same entity in a second modality of a lower resolution. The two data sets of different modalities are spatially registered to each other. From this information, a joint histogram of the values in the two datasets is computed to provide a raw analysis of how the intensities in the first dataset correspond to intensities in the second dataset. This is converted into a joint probability of possible intensities for the missing pixels in the low resolution dataset as a function of the intensities of the corresponding pixels in the high-resolution dataset to provide a very rough estimate of the intensities of the missing pixels in the low resolution dataset. Then, an objective function is defined over the set of possible new values that gives preference to datasets consistent with (1) the joint probability distributions, (2) the existing values in the low resolution dataset, and (3) smoothness throughout the data set. Finally, an annealing or similar iterative method is used to minimize the objective function and find an optimal solution over the entire dataset.

Growing community of inventors

San Jose, CA, United States of America

Dennis William Strelow