Microsoft Technology Licensing, LLC

Uber Technologies, Inc.

Microsoft Corporation

Sudipta Narayan Sinha

Richard S Szeliski

Raymond Kirk Price

Michael Bleyer

Christopher Douglas Edmonds

Ranveer Chandra

Deepak Vasisht

Eric Joel Stollnitz

Johannes P Kopf

Ashish Kapoor

Drew Edward Steedly

Blaise Aguera Y Arcas

Donald Allen Barnett

Markus Unger

David Maxwell Gedye

Sing Bing Kang

Neel Suresh Joshi

Anirudh Badam

Charles Lawrence Zitnick, Iii

Simon John Baker

Maneesh Agrawala

Marc Andre Leon Pollefeys

Amar Phanishayee

Zerina Kapetanovic

Krishnan Ramnath

Manohar Swaminathan

Joseph Michael Degol

Vasuki Narasimha Swamy

Xinxin Jin

Madhusudhan Gumbalapura Sudarshan

Akshit Kumar

Gireeja Ranade

Rohit Patil

Edward Hsiao

Pablo Alejandro Speciale

Rick Szeliski

David Nister

Michael F Cohen

Timothy Andrew Large

Nigel Stuart Keam

Vivek Pradeep

Geoffrey G Zweig

Vaibhav Thukral

Michael Edward Samples

Victor Manuel Fragoso Rojas

Christopher Stephen Messer

Matthew B Karr

Hyon Lim

Timo Pekka Pylvaenaeinen

Debadeepta Dey

Ondrej Miksik

Matt Uyttendaele

Shital Shah

Marc André Léon Pollefeys

Jae Yong Lee

Charles Zitnick

Mike Roberts

Minh Phuoc Vo

Varsha Hedau

Richard Roberts

Donald A Barnett

Johannes Lutz Schönberger

Tien Do

Johannes Kopf

Richard Stephen Szeliski

Johannes Peter Kopf

In various embodiments there is a method for camera localization within a scene. An image of a scene captured by the camera is input to a machine learning model, which has been trained for the particular scene to detect a plurality of 3D scene landmarks. The 3D scene landmarks are pre-specified in a pre-built map of the scene. The machine learning model outputs a plurality of predictions, each prediction comprising: either a 2D location in the image which is predicted to depict one of the 3D scene landmarks, or a 3D bearing vector, being a vector originating at the camera and pointing towards a predicted 3D location of one of the 3D scene landmarks. Using the predictions, an estimate of a position and orientation of the camera in the pre-built map of the scene is computed.

Camera localization

Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A 'system camera' refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

Low motion to photon latency rapid target acquisition

One example provides a computing system comprising a storage machine storing instructions executable by a logic machine to extract features from a source and target images to form source and target feature maps, form a correlation map comprising a plurality of similarity scores, form an initial correspondence map comprising initial mappings between pixels of the source feature map and corresponding pixels of the target feature map, refine the initial correspondence map by, for each of one or more pixels of the source feature map, for each of a plurality of candidate correspondences, inputting a four-dimensional patch into a trained scoring function, the trained scoring function being configured to output a correctness score, and selecting a refined correspondence based at least upon the correctness scores, and output a refined correspondence map comprising a refined correspondence for each of the one or more pixels of the source feature map.

Pixel correspondence via patch-based neighborhood consensus

Techniques for aligning images generated by two cameras are disclosed. This alignment is performed by computing a relative 3D orientation between the two cameras. A first gravity vector for a first camera and a second gravity vector for a second camera are determined. A first camera image is obtained from the first camera, and a second camera image is obtained from the second camera. A first alignment process is performed to partially align the first camera's orientation with the second camera's orientation. This process is performed by aligning the gravity vectors, thereby resulting in two degrees of freedom of the relative 3D orientation being eliminated. Visual correspondences between the two images are identified. A second alignment process is performed to fully align the orientations. This process is performed by using the identified visual correspondences to identify and eliminate a third degree of freedom of the relative 3D orientation.

Rapid target acquisition using gravity and north vectors

Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A 'system camera' refers to an integrated camera that is a part of an HMD. An 'external camera' is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

A system for continuous image alignment of separate cameras identifies a reference camera transformation matrix between a base reference camera pose and an updated reference camera pose. The system also identifies a match camera transformation matrix between a base match camera pose and an updated match camera pose and an alignment matrix based on visual correspondences between one or more reference frames captured by the reference camera and one or more match frames captured by the match camera. The system also generates a motion model configured to facilitate mapping of a set of pixels of a reference frame captured by the reference camera to a corresponding set of pixels of a match frame captured by the match camera based on the reference camera transformation matrix, the match camera transformation matrix, and the alignment matrix.

Systems and methods for continuous image alignment of separate cameras

A system for reducing a search area for identifying correspondences identifies an overlap region within a first match frame captured by a match camera. The overlap region includes one or more points of the first match frame that are associated with one or more same portions of an environment as one or more corresponding points of a first reference frame captured by a reference camera. The system obtains a second reference frame captured by the reference camera and a second match frame captured by the match camera. The system identifies a reference camera transformation matrix, and/or a match camera transformation matrix. The system defines a search area within the second match frame based on the overlap region and the reference camera transformation matrix and/or the match camera transformation matrix.

Systems and methods for reducing a search area for identifying correspondences between images

An imaging system that includes a camera mounted on an aerial platform, for example a balloon, allows a user to increase the longevity of the camera's battery by remote control. A user may capture imagery at a time scale of interest and desired power consumption by adjusting parameters for image capture by the camera. A user may adjust a time to capture an image, a time to capture a video, or a number of cycles per time period to capture one or more images as the aerial platform moves in a region of interest to change power consumption for imaging. The system also provides imaging alignment to account for unwanted movement of the aerial platform when moved in the region of interest. Additionally, a mounting device is provided that is simple and inexpensive, and that allows a camera to remain positioned in a desired position relative to the ground.

Low-cost, long-term aerial imagery

Computing devices and methods for estimating a pose of a user computing device are provided. In one example a 3D map comprising a plurality of 3D points representing a physical environment is obtained. Each 3D point is transformed into a 3D line that passes through the point to generate a 3D line cloud. A query image of the environment captured by a user computing device is received, the query image comprising query features that correspond to the environment. Using the 3D line cloud and the query features, a pose of the user computing device with respect to the environment is estimated.

Device pose estimation using 3D line clouds

One or more techniques and/or systems are provided for ordering images for panorama stitching and/or for providing a focal point indicator for image capture. For example, one or more images, which may be stitched together to create a panorama of a scene, may be stored within an image stack according to one or more ordering preferences, such as where manually captured images are stored within a first/higher priority region of the image stack as compared to automatically captured images. One or more images within the image stack may be stitched according to a stitching order to create the panorama, such as using images in the first region for a foreground of the panorama. Also, a current position of a camera may be tracked and compared with a focal point of a scene to generate a focal point indicator to assist with capturing a new/current image of the scene.

Image capture and ordering

Stereo image reconstruction can be achieved by fusing a plurality of proposal cost volumes computed from a pair of stereo images, using a predictive model operating on pixelwise feature vectors that include disparity and cost values sparsely sampled form the proposal cost volumes to compute disparity estimates for the pixels within the image.

Fusing disparity proposals in stereo matching

A method for estimating a camera pose includes recognizing a three-dimensional (3D) map representing a physical environment, the 3D map including 3D map features defined as 3D points. An obfuscated image representation is received, the representation derived from an original unobfuscated image of the physical environment captured by a camera. The representation includes a plurality of obfuscated features, each including (i) a two-dimensional (2D) line that passes through a 2D point in the original unobfuscated image at which an image feature was detected, and (ii) a feature descriptor that describes the image feature associated with the 2D point that the 2D line of the obfuscated feature passes through. Correspondences are determined between the obfuscated features and the 3D map features of the 3D map of the physical environment. Based on the determined correspondences, a six degree of freedom pose of the camera in the physical environment is estimated.

Camera pose estimation using obfuscated features

Examples of the present disclosure relate to generating optimal scanning trajectories for 3D scenes. In an example, a moveable camera may gather information about a scene. During an initial pass, an initial trajectory may be used to gather an initial dataset. In order to generate an optimal trajectory, a reconstruction of the scene may be generated based on the initial data set. Surface points and a camera position graph may be generated based on the reconstruction. A subgradient may be determined, wherein the subgradient provides an additive approximation for the marginal reward associated with each camera position node in the camera position graph. The subgradient may be used to generate an optimal trajectory based on the marginal reward of each camera position node. The optimal trajectory may then be used by to gather additional data, which may be iteratively analyzed and used to further refine and optimize subsequent trajectories.

Optimal scanning trajectories for 3D scenes

Growing community of inventors

Kirkland, WA, United States of America

Sudipta Narayan Sinha