Microsoft Technology Licensing, LLC

Facebook, Inc.

Meta Platforms, Inc.

Johannes Peter Kopf

David Chiyuan Chu

Eduardo Alberto Cuervo Laffaye

Alastair Wolman

Sergey Grizan

Yury Degtyarev

Kyungmin Lee

Michael F Cohen

Richard S Szeliski

John Samuel Barnett

Bernhard Reinert

Daniel Lischinski

Matthieu Tony Uyttendaele

Suhib Fakhri Mahmod Alsisan

Jason George McHugh

Piotr Dollar

Fabian Langguth

Brian Dolhansky

Andrei Chtcherbatchenko

Bo Yin

Lars Peter Johannes Hedman

Shi Chen

Francis Yunfeng Ge

In one embodiment, a method includes generating an outputted sequence of warped images from a captured sequence of images. Using this captured sequence of images, a computing system may determine one or more three-dimensional locations of object features and a corresponding camera position for each image in the captured sequence of images. Utilizing the camera positions for each image, the computing system may determine a view path representing the perspective of a virtual camera. The computing system may identify one or more virtual camera positions for the virtual camera located on the view path, and subsequently warp one or more images from the sequence of captured images to represent the perspective of the virtual camera at each of the respective virtual camera positions. This results in a sequence of warped images that may be outputted for viewing and interaction on a client device.

Outputting warped images from captured video data

A server device and method are provided for use in predictive server-side rendering of scenes based on client-side user input. The server device may include a processor and a storage device holding instructions for an application program executable by the processor to receive, at the application program, a current navigation input in a stream of navigation inputs from a client device over a network, calculate a predicted future navigation input based on the current navigation input and a current application state of the application program, render a future scene based on the predicted future navigation input to a rendering surface, and send the rendering surface to the client device over the network.

Predictive server-side rendering of scenes

Systems, methods, and non-transitory computer readable media are configured to detect a concept reflected in a first media content item to which a user is provided access. It is determined that the concept has a threshold level of relevance to the user. The concept is associated with an element that upon selection causes a transition to a second media content item to which the user is provided access, the second media content item reflecting the concept. The element is presented in the first media content item for the user.

Systems and methods to transition between media content items

In one embodiment, a method includes generating depth map for a reference image and generating a three-dimensional (3D) model for a plurality of objects in the reference image based on the depth map. The method additionally includes determining, out of the objects in the 3D model, a background object having a boundary adjacent to a foreground object. The method also includes determining that at least a portion of a surface of the background object is hidden by the foreground object and extending, in the 3D model, the surface of the background object to include the portion hidden by the foreground object. The method further includes in-paint pixels of the extended surface of the background object with pixels that approximate the portion of the surface of the background object hidden by the foreground object.

Image object extraction and in-painting hidden surfaces for modified viewpoint rendering

A head mounted display device including a processor configured to compute a rendered rendering surface of a predicted scene having a predicted user viewpoint, the predicted user viewpoint being a prediction of a viewpoint that a user will have at a point in time that was predicted for the user of the head mounted display device prior to the point in time, receive, from the user input device, a subsequent user navigation input near the point in time in the stream of user input, determine an actual user viewpoint based on the subsequent user navigation input, determine a user viewpoint misprediction based on the predicted user viewpoint and the actual user viewpoint, and reconstruct a viewport for the actual user viewpoint from the rendered rendering surface.

Reconstructing viewport upon user viewpoint misprediction

Systems, methods, and non-transitory computer-readable media can generate an initial alpha mask for an image based on machine learning techniques. A plurality of uncertain pixels is defined in the initial alpha mask. For each uncertain pixel in the plurality of uncertain pixels, a binary value is assigned based on a nearest certain neighbor determination.

Systems and methods for image matting

An image processing system generates 360-degree stabilized videos with higher robustness, speed, and smoothing ability using a hybrid 3D-2D stabilization model. The image processing system first receives an input video data (e.g., a 360-degree video data) for rotation stabilization. After tracking feature points through the input video data, the image processing system determines key frames and estimates rotations of key frames using a 3D reasoning based on the tracked feature points. The image processing system also optimizes inner frames between key frames using a 2D analysis based on the estimated key frame rotation. After the 3D reasoning and the 2D analysis, the image processing system may reapply a smoothed version of raw rotations to preserve desirable rotations included in the original input video data, and generates a stabilized version of the input video data (e.g., a 360-degree stabilized video).

Video stabilization system for 360-degree video data

A first user input is received when a client program executed by a client computing device is in a first state. The first user input is sent to a server computing device to render a view of a virtual scene. A state change from the first state in the client program due to a second user input or a program event is identified. One or more gaps in a server-rendered current view due to the state change are determined. A rendering of the one or more gaps is selected from among the server-rendered current view, a server-rendered predicted view and one or more prior-rendered views. A current view is rendered using a simplified model of the virtual scene by rendering the one or more gaps from the selected rendering. The current rendered view is visually presented via a display of the client computing device.

View rendering from multiple server-side renderings

To enable better sharing and preservation of immersive experiences, a graphics system reconstructs a three-dimensional scene from a set of images of the scene taken from different vantage points. The system processes each image to extract depth information therefrom and then stitches the images (both color and depth information) into a multi-layered panorama that includes at least front and back surface layers. The front and back surface layers are then merged to remove redundancies and create connections between neighboring pixels that are likely to represent the same object, while removing connections between neighboring pixels that are not. The resulting layered panorama with depth information can be rendered using a virtual reality (VR) system, a mobile device, or other computing and display platforms using standard rendering techniques, to enable three-dimensional viewing of the scene.

Three-dimensional scene reconstruction from set of two dimensional images for consumption in virtual reality

Various technologies described herein pertain to generation of an output hyper-lapse video from an input video. A smoothed camera path can be computed based upon the input video. Further, output camera poses can be selected from the smoothed camera path for output frames of the output hyper-lapse video. One or more selected input frames from the input video can be chosen for an output frame. The selected input frames can be chosen based at least in part upon an output camera pose for the output frame. Moreover, the selected input frames can be combined to render the output frame. Choosing selected input frames from the input video and combining the selected input frames can be performed for each of the output frames of the output hyper-lapse video.

Growing community of inventors

Hamburg, Germany

Johannes Peter Kopf