Google Inc.

Microsoft Technology Licensing, LLC

University of Washington

Steven Maxwell Seitz

Richard S Szeliski

Robert J Anderson

Andrew Ian Russell

Sing Bing Kang

Andrew Gerrit Huibers

Carlos Hernandez Esteban

David Robert Gallup

Keith Noah Snavely

Andrew Gerritt Huibers

Andrew Lookingbill

Daniel Goldman

Qi Shan

Rahul Garg

Andrew Ofstad

Sameer Agarwal

Jason Lawrence

Li Zhang

Matthew Robert Simpson

Maneesh Agrawala

Ricardo Martin Brualla

Robert E Carroll

Irena Kemelmaher

Supasorn Suwajanakorn

Alexandros Andre Chaaraoui

Fu Yu

Eric J Seibel

Neel Suresh Joshi

Joshua L Weaver

Jonah Jones

Liyong Chen

Peter-Pike Johannes Sloan

Erik Hubert Dolly Goossens

Shuchang Zhou

Christopher Edward Hoover

Aditya Sankar

Matthew Thomas Valente

Michael Krainin

Yasutaka Furukawa

Changchang Wu

Riley Adams

Jianxiong Xiao

Jeffrey Thomas Prouty

Yunpeng Li

Simon Fuhrmann

Pierre Georgel

An example telepresence terminal includes a display, an image sensor, an infrared emitter, and an infrared depth sensor. The terminal may determine image data using visible light emitted by the infrared emitter and captured by the image sensor and determine depth data using infrared light captured by the infrared depth sensor. The terminal may also communicate the depth data and the image data to a remote telepresence terminal and receive remote image data and remote depth data. The terminal may also generate a first display image using the lenticular display based on the remote image data that is viewable from a first viewing location and generate a second display image using the lenticular display based on the remote image data and the remote depth data that is viewable from a second viewing location.

Three-dimensional telepresence system

An example telepresence terminal includes a lenticular display, an image sensor, an infrared emitter, and an infrared depth sensor. The terminal may determine image data using visible light emitted by the infrared emitter and captured by the image sensor and determine depth data using infrared light captured by the infrared depth sensor. The terminal may also communicate the depth data and the image data to a remote telepresence terminal and receive remote image data and remote depth data. The terminal may also generate a first display image using the lenticular display based on the remote image data that is viewable from a first viewing location and generate a second display image using the lenticular display based on the remote image data and the remote depth data that is viewable from a second viewing location.

Aspects of the disclosure relate generally to generating depth data from a video. As an example, one or more computing devices may receive an initialization request for a still image capture mode. After receiving the request to initialize the still image capture mode, the one or more computing devices may automatically begin to capture a video including a plurality of image frames. The one or more computing devices track features between a first image frame of the video and each of the other image frames of the video. Points corresponding to the tracked features may be generated by the one or more computing devices using a set of assumptions. The assumptions may include a first assumption that there is no rotation and a second assumption that there is no translation. The one or more computing devices then generate a depth map based at least in part on the points.

Depth map generation

Aspects of the disclosure relate to capturing panoramic images using a computing device. For example, the computing device may record a set of video frames and tracking features each including one or more features that appear in two or more video frames of the set of video frames within the set of video frames may be determined. A set of frame-based features based on the displacement of the tracking features between two or more video frames of the set of video frames may be determined by the computing device. A set of historical feature values based on the set of frame-based features may also be determined by the computing device. The computing device may determine then whether a user is attempting to capture a panoramic image based on the set of historical feature values. In response, the computing device may capture a panoramic image.

Automatic detection of panoramic gestures

Systems and methods for capturing omnistereo content for a mobile device may include receiving an indication to capture a plurality of images of a scene, capturing the plurality of images using a camera associated with a mobile device and displaying on a screen of the mobile device and during capture, a representation of the plurality of images and presenting a composite image that includes a target capture path and an indicator that provides alignment information corresponding to a source capture path associated with the mobile device during capture of the plurality of images. The system may detect that a portion of the source capture path does not match a target capture path. The system can provide an updated indicator in the screen that may include a prompt to a user of the mobile device to adjust the mobile device to align the source capture path with the target capture path.

Omnistereo capture for mobile devices

Systems and methods are related to a camera rig and generating stereoscopic panoramas from captured images for display in a virtual reality (VR) environment.

Camera rig and stereoscopic image capture

Systems and methods are described for defining a set of images based on captured images, receiving a viewing direction associated with a user of a virtual reality (VR) head mounted display, receiving an indication of a change in the viewing direction. The methods further include configuring, a re-projection of a portion of the set of images, the re-projection based at least in part on the changed viewing direction and a field of view associated with the captured images, and converting the portion from a spherical perspective projection into a planar perspective projection, rendering by the computing device and for display in the VR head mounted display, an updated view based on the re-projection, the updated view configured to correct distortion and provide stereo parallax in the portion, and providing, to the head mounted display, the updated view including a stereo panoramic scene corresponding to the changed viewing direction.

Capture and render of panoramic virtual reality content

An indoor scene capture system is provided that, with a handheld device with a camera, collects videos of rooms, spatially indexes the frames of the videos, marks doorways between rooms, and collects videos of transitions from room to room via doorways. The indoor scene capture system may assign a direction to at least some of the frames based on the angle of rotation as determined by an inertial sensor (e.g., gyroscope) of the handheld device. The indoor scene capture system marks doorways within the frames of the videos. For each doorway between rooms, the indoor scene capture system collects a video of transitioning through the doorway as the camera moves from the point within a room through the doorway to a point within the adjoining room.

Indoor scene capture system

Methods, systems, and articles of manufacture for generating a panoramic image of a long scene, are disclosed. These include, fitting a plurality of planes to 3D points associated with input images of portions of the long scene, where one or more respective planes are fitted to each of a ground surface, a dominant surface, and at least one of one or more foreground objects and one or more background objects in the long scene, and where distances from the 3D points to the fitted planes are substantially minimized. These also include, selecting, for respective one or more pixels in the panoramic image of the long scene, one of the input images and one of the fitted planes such that a distance is substantially minimized from the selected one of the fitted planes to a surface corresponding to the respective one or more pixels and occlusion of the respective one or more pixels is reduced in the selected one of the input images; and stitching the panoramic image of the long scene by projecting, for the respective one or more pixels in the panoramic image of the long scene, the selected one of the input images using the selected one of the fitted planes into the virtual camera.

Multiplane panoramas of long scenes

An exemplary method includes prompting a user to capture video data at a location. The location is associated with navigation directions for the user. Information representing visual orientation and positioning information associated with the captured video data is received by one or more computing devices, and a stored data model representing a 3D geometry depicting objects associated with the location is accessed. Between corresponding images from the captured video data and projections of the 3D geometry, one or more candidate change regions are detected. Each candidate change region indicates an area of visual difference between the captured video data and projections. When it is detected that a count of the one or more candidate change regions is below a threshold, the stored model data is updated with at least part of the captured video data based on the visual orientation and positioning information associated with the captured video data.

3D model updates using crowdsourced video

Systems and methods for navigating an imagery graph are provided. In some aspects, a first image is provided for display, where the first image corresponds to a first image node within an imagery graph, where the imagery graph comprises image nodes corresponding to images from a plurality of different imagery types, and where each image node in the imagery graph is associated with geospatial data. An indication of a selection of a predetermined region within the first image is received, where the predetermined region is associated with a position in the first image that corresponds to geospatial data associated a second image node within the imagery graph. A second image corresponding to the second image node is provided for display in response to the indication of the selection of the predetermined region.

Navigating visual data associated with a point of interest

Systems and methods for the generation of depth data for a scene using images captured by a camera-enabled mobile device are provided. According to a particular implementation of the present disclosure, a reference image can be captured of a scene with an image capture device, such as an image capture device integrated with a camera-enabled mobile device. A short video or sequence of images can then be captured from multiple different poses relative to the reference scene. The captured image and video can then be processed using computer vision techniques to produce an image with associated depth data, such as an RGBZ image.

Computer vision algorithm for capturing and refocusing imagery

Growing community of inventors

Seattle, WA, United States of America

Steven Maxwell Seitz