Google Inc.

Hugues Hoppe

Ricardo Martin Brualla

Daniel Goldman

Rohit Kumar Pandey

Sameh Khamis

Sebastian Sylvan

Noah Snavely

Harris Samuel Nover

Moustafa Meshry

David Chu

True Price

Grigory Javadyan

Egor Yusov

Kelvin Ritland

Bernhard Reinert

Lynn Tsai

Lars Peter Johannes Hedman

A method includes obtaining a first frame of image data including a plurality of pixels. Each pixel of the plurality of pixels is associated with a respective color value and a respective depth value. The first frame renders a scene from a first point of view. The method includes generating a three-dimensional (3D) polygon mesh using the plurality of pixels and the respective depth values. The 3D polygon mesh includes a plurality of portions. Each respective portion defines a respective plurality of vertices defining a respective mesh density representative of a density of the respective plurality of vertices. Each vertex is associated with a corresponding pixel of the first frame of image data. The method includes generating a second frame of image data via reprojection using the generated 3D polygon mesh. The second frame of image data has a second point of view different from the first point of view.

Adaptive mesh reprojection for low latency 6DOF rendering

Systems and methods are described for receiving a plurality of input images, a plurality of depth images, and a plurality of view parameters for generating a virtual view of a target subject. The systems and methods may generate a plurality of warped images based on the plurality of input images, the plurality of view parameters, and at least one of the plurality of depth images. In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to a neural network, the systems and methods may receive, from the neural network, blending weights for assigning color to pixels of the virtual view of the target subject and may generate, based on the blending weights and the virtual view, a synthesized image according to the view parameters.

Neural blending for novel view synthesis

According to an aspect, a method for neural rerendering includes obtaining a three-dimensional (3D) model representing a scene of a physical space, where the 3D model is constructed from a collection of input images, rendering an image data buffer from the 3D model according to a viewpoint, where the image data buffer represents a reconstructed image from the 3D model, receiving, by a neural rerendering network, the image data buffer, receiving, by the neural rerendering network, an appearance code representing an appearance condition, and transforming, by the neural rerendering network, the image data buffer into a rerendered image with the viewpoint of the image data buffer and the appearance condition specified by the appearance code.

Neural rerendering from 3D models

Techniques of rendering images includes generating signed distance values (SDVs) along a ray from a specified viewpoint in terms of projected distances along that ray from given depth images. For each pixel in an image of from the perspective of the specified viewpoint, a ray is traced into the three-dimensional scene represented by the image. An iterative step is performed along the ray, obtaining in each iteration a three-dimensional world-space point p. The result is the signed distance sas measured from depth view D. If the absolute value of the signed distance sis greater than some truncation threshold parameter, the signed distance sis replaced by a special undefined value. The defined signed-distance values are aggregated to obtain an overall signed distance s. Finally, the roots or zero set (isosurface) of the signed distance field is determined.

Image-based geometric fusion of multiple depth images using ray casting

Techniques of compressing color video images include computing a delta quantization parameter (&#x394;QP) for the color images based on a similarity between the depth image surface normal and the view direction associated with a color image. For example, upon receiving a frame having an image with multiple color and depth images, a computer finds a depth image that is closest in orientation to a color image. For each pixel of that depth image, the computer generates a blend weight based on an orientation of a normal to a position of the depth image and the viewpoints from which the plurality of color images were captured. The computer then generates a value of &#x394;QP based on the blend weight and determines a macroblock of color image corresponding to the position, the macroblock being associated with the value of &#x394;QP for the pixel.

Spatially adaptive video compression for multiple streams of color and depth

Growing community of inventors

Mercer Island, WA, United States of America

Hugues Hoppe