Meta Platforms Technologies, LLC

Meta Platforms, Inc.

Johannes Peter Kopf

John Samuel Barnett

Jia-Bin Huang

Changil Kim

Xuejian Rong

Michael Zollhoefer

Wenqi Xian

Ocean Quigley

Tuotuo Li

Benjamin Attal

Michael F Cohen

Bryce Alan Evans

In particular embodiments, a computing system may access a set of training images for training a neural light field network for view synthesis. Using each training image, the computing system may train the neural light field network by casting, for each pixel of the training image, a ray into a three-dimensional (3D) space, the ray including integrated radiance along the ray, mapping first ray coordinates of the ray into an embedding network, transforming, using the embedding network, the first ray coordinates into second ray coordinates, applying positional encoding to second ray coordinates, generating, using the neural light field network, a predicted color value for the pixel based on positionally encoded second ray coordinates, comparing the predicted color value for the pixel with a ground-truth color value for the pixel, and updating the neural light field network and the embedding network based on comparison.

Learning neural light fields with ray-space embedding networks

In particular embodiments, a computing system may receive a video comprising a plurality of image frames. The system may generate, for each image frame in the video, an initial depth map using a machine-learning model. The system may compute a misalignment error indicating depth misalignments in initial depth maps using a reprojection technique. The system may generate, for each image frame in the video, an optimized camera pose and a flexible deformation spline associated with the image frame to minimize the misalignment error. The system may generate, for each image frame in the video, a refined depth map by adjusting the initial depth map associated with the frame using the flexible deformation spline associated with the image frame.

Robust consistent video depth estimation

In one embodiment, a method includes accessing multiple 3D photos to be concurrently displayed through multiple frames positioned in a virtual space, each of the of 3D photos having an optimal viewing point in the virtual space and determining a reference point based on a head pose of a viewer within the virtual space. The method may further include adjusting each 3D photo by rotating the 3D photo so that the optimal viewing point of the 3D photo points at the reference point, translating the rotated 3D photo toward the reference point, and non-uniformly scaling the rotated and translated 3D photo based on a scaling factor determined using the reference point and a position of the frame through which the 3D photo is to be viewed. The method may further include rendering an image comprising the adjusted multiple 3D photos as seen through the multiple frames.

Viewing 3D photo galleries in VR

In one embodiment, a computing system may determine a view position, a view direction, and a time with respect to a scene. The system may access a spatiotemporal representation of the scene generated based on (1) a monocular video including images each capturing at least a portion of the scene at a corresponding time and (2) depth values of the portion of the scene captured by each image. The system may generate an image based on the view position, the view direction, the time, and the spatiotemporal representation. A pixel value of the image corresponding to the view position may be determined based on volume densities and color values at sampling locations along the view direction and at the time in the spatiotemporal representation. The system may output the image to the display, representing the scene at the time as viewed from the view position and in the view direction.

Space-time representation of dynamic scenes

Growing community of inventors

Medina, WA, United States of America

Johannes Peter Kopf