Microsoft Technology Licensing, LLC

University of Washington

Facebook, Inc.

Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Meta Platforms, Inc.

Richard S Szeliski

Matthew Uyttendaele

Sing Bing Kang

Charles Lawrence Zitnick, Iii

Simon Winder

Heung-Yeung Shum

Michael F Cohen

Sudipta Narayan Sinha

Drew Edward Steedly

Kenneth Paul Hinckley

Michael Jack Sinclair

Matthew Conway

Erik J Hanson

Steven Mark Drucker

Donald James Lindsay

Gary William Flake

Blaise Aguera Y Arcas

Brett David Brewer

Stephen L Lawler

Simon John Baker

Karim Taji Farouki

Adam P Sheppard

Baining Guo

Steven Maxwell Seitz

Jeffrey Jon Weir

David H Salesin

Lifeng Wang

Keith Noah Snavely

Tomasz Kasperkiewicz

Nicholas P Wilt

Neel Suresh Joshi

Maneesh Agrawala

Johannes Peter Kopf

Padmanabhan Anandan

Padmananbhan Anandan

Shmuel Avidan

Arno Sch&ouml;dl

Mei Han

Charles L Zitnick

David James Geraghty

Andreas Georgiou

Julio Estrada

Eric Joel Stollnitz

Steven Jacob Gortler

Tomasz S M Kasperkiewicz

Johannes P Kopf

Eric Paul Bennett

Krishnan Ramnath

Chris Pal

Edward Hsiao

Desney S Tan

Shahram Izadi

Patrice Y Simard

Josh D Benaloh

Eric H Rudolph

Hugues Herve Hoppe

Patrick Markus Baudisch

Aseem Agarwala

Antonio Criminisi

Christopher Bruce Weare

Ce Liu

Geoffrey G Zweig

Georg F Petschnigg

Avronil Bhattacharjee

Adam G Kirk

Iulian Doroftei Calinov

David W Steinkraus

Christoph Rhemann

Oliver Arthur Whyte

Philip H S Torr

Jeremy S De Bonet

Simon A J Winder

Julien Couvreur

Fabian Langguth

Ronald Keith Logan

Ian Andrew Buck

Radek Grzeszczuk

Andrei Chtcherbatchenko

Daniel Scharstein

Matthew Brown

Charles Zitnick

Suhib Fakhri Mahmod Alsisan

Bo Yin

Blaise H Aguera

Jie Sun

Ashley Michelle Eden

Shi Chen

Francis Yunfeng Ge

Varsha Hedau

Polina Golland

Richard Roberts

Blaise Aguera Y Aguera

Xiangyang Lan

Lars Peter Johannes Hedman

Johannes Kopf

An artificial reality system is configured to more accurately and efficiently construct a 3D virtual representation of a real-world environment from a set of 2D images. The system identifies points and/or lines within the images that define a plane along an orientation and then performs a planar sweep along a perpendicular path to identify surfaces in which the plane intersects with multiple points. Points that appear to be in the same plane are 'collapsed' into a cohesive plane to conserve processing power by estimating and/or storing parameters for the cohesive plane, rather than all of the individual 3D points. In this way, the system also 'averages out' random variation in the planar surface that would otherwise result from random noise in the estimation of the individual 3D points. The system may then generate a 3D map from a constrained alignment of all of the identified planes.

Artificial reality system with 3D environment reconstruction using planar constraints

In one embodiment, a method includes generating an outputted sequence of warped images from a captured sequence of images. Using this captured sequence of images, a computing system may determine one or more three-dimensional locations of object features and a corresponding camera position for each image in the captured sequence of images. Utilizing the camera positions for each image, the computing system may determine a view path representing the perspective of a virtual camera. The computing system may identify one or more virtual camera positions for the virtual camera located on the view path, and subsequently warp one or more images from the sequence of captured images to represent the perspective of the virtual camera at each of the respective virtual camera positions. This results in a sequence of warped images that may be outputted for viewing and interaction on a client device.

Outputting warped images from captured video data

The subject disclosure is directed towards projecting light in a pattern in which the pattern contains components (e.g., spots) having different intensities. The pattern may be based upon a grid of initial points associated with first intensities and points between the initial points with second intensities, and so on. The pattern may be rotated relative to cameras that capture the pattern, with captured images used active depth sensing based upon stereo matching of dots in stereo images.

Intensity-modulated light pattern for active stereo

In one embodiment, a method includes accessing an image of a physical world scene, detecting a number of straight-line segments in the accessed image, identifying a first vanishing point and a second vanishing point in the image, where each vanishing point corresponds to a sub-set of the number of straight-line segments, and where the first vanishing point and the second vanishing point are orthogonal, identifying a planar region in the accessed image represented by one or more straight-line segments associated with the first vanishing point and one or more straight-line segments associated with the second vanishing point, transforming a virtual object associated with the planar region based on one or more properties associated with the planar region, and displaying the transformed virtual object over the image.

Identifying planes in artificial reality systems

To enable better sharing and preservation of immersive experiences, a graphics system reconstructs a three-dimensional scene from a set of images of the scene taken from different vantage points. The system processes each image to extract depth information therefrom and then stitches the images (both color and depth information) into a multi-layered panorama that includes at least front and back surface layers. The front and back surface layers are then merged to remove redundancies and create connections between neighboring pixels that are likely to represent the same object, while removing connections between neighboring pixels that are not. The resulting layered panorama with depth information can be rendered using a virtual reality (VR) system, a mobile device, or other computing and display platforms using standard rendering techniques, to enable three-dimensional viewing of the scene.

Three-dimensional scene reconstruction from set of two dimensional images for consumption in virtual reality

Various technologies described herein pertain to generation of an output hyper-lapse video from an input video. A smoothed camera path can be computed based upon the input video. Further, output camera poses can be selected from the smoothed camera path for output frames of the output hyper-lapse video. One or more selected input frames from the input video can be chosen for an output frame. The selected input frames can be chosen based at least in part upon an output camera pose for the output frame. Moreover, the selected input frames can be combined to render the output frame. Choosing selected input frames from the input video and combining the selected input frames can be performed for each of the output frames of the output hyper-lapse video.

Hyper-lapse video through time-lapse and stabilization

The subject disclosure is directed towards a high resolution, high frame rate, robust stereo depth system. The system provides depth data in varying conditions based upon stereo matching of images, including actively illuminated IR images in some implementations. A clean IR or RGB image may be captured and used with any other captured images in some implementations. Clean IR images may be obtained by using a notch filter to filter out the active illumination pattern. IR stereo cameras, a projector, broad spectrum IR LEDs and one or more other cameras may be incorporated into a single device, which may also include image processing components to internally compute depth data in the device for subsequent output.

Depth imaging system based on stereo vision and infrared radiation

The claimed subject matter provides for systems and/or methods for identification of instances of an object of interest in 2D images by creating a database of 3D curve models of each desired instance and comparing an image of an object of interest against such 3D curve models of instances. The present application describes identifying and verifying the make and model of a car from a possibly single image&#x2014;after the models have been populated with training data of test images of many makes and models of cars. In one embodiment, an identification system may be constructed by generating a 3D curve model by back-projecting edge points onto a visual hull reconstruction from silhouettes of an instance. The system and methods employ chamfer distance and orientation distance provides reasonable verification performance, as well as an appearance model for the taillights of the car to increase the robustness of the system.

Object identification using 3-D curve matching

A mobile device having the capability of performing real-time location recognition with assistance from a server is provided. The approximate geophysical location of the mobile device is uploaded to the server. Based on the mobile device's approximate geophysical location, the server responds by sending the mobile device a message comprising a classifier and a set of feature descriptors. This can occur before an image is captured for visual querying. The classifier and feature descriptors are computed during an offline training stage using techniques to minimize computation at query time. The classifier and feature descriptors are used to perform visual recognition in real-time by performing the classification on the mobile device itself.

Location-aided recognition

Images are analyzed within a 3D environment that is generated based on spatial relationships of the images and that allows users to experience the images in the 3D environment. Image analysis may include ranking images based on user viewing information, such as the number of users who have viewed an image and how long an image was viewed. Image analysis may further include analyzing the spatial density of images within a 3D environment to determine points of user interest.

Analysis of images located within three-dimensional environments

The claimed subject matter provides a system and/or a method that facilitates servicing a portion of a trade card via a web service. A web service can provide a portion of data to enhance a trade card, wherein the portion of data is at least one of a portion of trade card document-specific data, an intelligent gadget, or a feed driven component. A build component can leverage the web service to utilize the portion of data with the trade card.

Trade card services

A document that includes a representation of a two-dimensional (2-D) image may be obtained. A selection indicator indicating a selection of at least a portion of the 2-D image may be obtained. A match correspondence may be determined between the selected portion of the 2-D image and a three-dimensional (3-D) image object stored in an object database, the match correspondence based on a web crawler analysis result. A 3-D rendering of the 3-D image object that corresponds to the selected portion of the 2-D image may be initiated.

Three-dimensional object browsing in documents

Methods, systems, and computer-readable media for reconstruction a three-dimensional scene from a collection of two-dimensional images are provided. A computerized reconstruction system executes computer vision algorithms on the collection of two-dimensional images to identify candidate planes that are used to model visual characteristics of the environment depicted in the two-dimensional images. The computer vision algorithms may minimize an energy function that represents the relationships and similarities among features of the two-dimensional images to assign pixels of the two dimensional images to planes in the three dimensional scene. The three-dimensional scene is navigable and depicts viewpoint transitions between multiple two-dimensional images.

Piecewise planar reconstruction of three-dimensional scenes

Growing community of inventors

Bellevue, WA, United States of America

Richard S Szeliski