Adobe, Inc.

Portland State University

Simon Niklaus

Oliver Wang

Jianming Zhang

Feng Liu

Mai Long

Su Chen

Long Mai

Wei Yin

Zhe Lin

Jimei Yang

Zhengqi Li

Soo Ye Kim

Ping Hu

Yifei Fan

Jiawen Chen

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate refined depth maps of digital images utilizing digital segmentation masks. In particular, in one or more embodiments, the disclosed systems generate a depth map for a digital image utilizing a depth estimation machine learning model, determine a digital segmentation mask for the digital image, and generate a refined depth map from the depth map and the digital segmentation mask utilizing a depth refinement machine learning model. In some embodiments, the disclosed systems generate first and second intermediate depth maps using the digital segmentation mask and an inverse digital segmentation mask and merger the first and second intermediate depth maps to generate the refined depth map.

Utilizing machine learning models to generate refined depth maps with segmentation mask guidance

Methods, systems, and storage media are described for motion estimation in video frame interpolation. Disclosed embodiments use feature pyramids as image representations for motion estimation and seamlessly integrates them into a deep neural network for frame interpolation. A feature pyramid is extracted for each of two input frames. These feature pyramids are wrapped together with the input frames to the target temporal position according to the inter-frame motion estimated via optical flow. A frame synthesis network is used to predict interpolation results from the pre-warped feature pyramids and input frames. The feature pyramid extractor and the frame synthesis network are jointly trained for the task of frame interpolation. An extensive quantitative and qualitative evaluation demonstrates that the described embodiments utilizing feature pyramids enables robust, high-quality video frame interpolation. Other embodiments may be described and/or claimed.

Feature pyramid warping for video frame interpolation

Digital image synthesis techniques are described that leverage splatting, i.e., forward warping. In one example, a first digital image and a first optical flow are received by a digital image synthesis system. A first splat metric and a first merge metric are constructed by the digital image synthesis system that defines a weighted map of respective pixels. From this, the digital image synthesis system produces a first warped optical flow and a first warp merge metric corresponding to an interpolation instant by forward warping the first optical flow based on the splat metric and the merge metric. A first warped digital image corresponding to the interpolation instant is formed by the digital image synthesis system by backward warping the first digital image based on the first warped optical flow.

Splatting-based digital image synthesis

Digital synthesis techniques are described to synthesize a digital image at a target time between a first digital image and a second digital image. To begin, an optical flow generation module is employed to generate optical flows. The digital images and optical flows are then received as an input by a motion refinement system. The motion refinement system is configured to generate data describing many-to-many relationships mapped for pixels in the plurality of digital images and reliability scores of the many-to-many relationships. The reliability scores are then used to resolve overlaps of pixels that are mapped to a same location by a synthesis module to generate a synthesized digital image.

Many-to-many splatting-based digital image synthesis

Embodiments of the technology described herein, provide a view and time synthesis of dynamic scenes captured by a camera. The technology described herein represents a dynamic scene as a continuous function of both space and time. The technology may parameterize this function with a deep neural network (a multi-layer perceptron (MLP)), and perform rendering using volume tracing. At a very high level, a dynamic scene depicted in the video may be used to train the MLP. Once trained, the MLP is able to synthesize a view of the scene at a time and/or camera pose not found in the video through prediction. As used herein, a dynamic scene comprises one or more moving objects.

View synthesis of a dynamic scene

Embodiments are disclosed for video image interpolation. In some embodiments, video image interpolation includes receiving a pair of input images from a digital video, determining, using a neural network, a plurality of spatially varying kernels each corresponding to a pixel of an output image, convolving a first set of spatially varying kernels with a first input image from the pair of input images and a second set of spatially varying kernels with a second input image from the pair of input images to generate filtered images, and generating the output image by performing kernel normalization on the filtered images.

Optimization of adaptive convolutions for video frame interpolation

This disclosure describes one or more implementations of a depth prediction system that generates accurate depth images from single input digital images. In one or more implementations, the depth prediction system enforces different sets of loss functions across mix-data sources to generate a multi-branch architecture depth prediction model. For instance, in one or more implementations, the depth prediction model utilizes different data sources having different granularities of ground truth depth data to robustly train a depth prediction model. Further, given the different ground truth depth data granularities from the different data sources, the depth prediction model enforces different combinations of loss functions including an image-level normalized regression loss function and/or a pair-wise normal loss among other loss functions.

Generating depth images utilizing a machine-learning model built from mixed digital image sources and multiple loss function sets

Systems, methods, and computer-readable media for context-aware synthesis for video frame interpolation are provided. Bidirectional flow may be used in combination with flexible frame synthesis neural network to handle occlusions and the like, and to accommodate inaccuracies in motion estimation. Contextual information may be used to enable frame synthesis neural network to perform informative interpolation. Optical flow may be used to provide initialization for interpolation. Other embodiments may be described and/or claimed.

Context-aware synthesis for video frame interpolation

Systems, methods, and computer-readable media for context-aware synthesis for video frame interpolation are provided. A convolutional neural network (ConvNet) may, given two input video or image frames, interpolate a frame temporarily in the middle of the two input frames by combining motion estimation and pixel synthesis into a single step and formulating pixel interpolation as a local convolution over patches in the input images. The ConvNet may estimate a convolution kernel based on a first receptive field patch of a first input image frame and a second receptive field patch of a second input image frame. The ConvNet may then convolve the convolutional kernel over a first pixel patch of the first input image frame and a second pixel patch of the second input image frame to obtain color data of an output pixel of the interpolation frame. Other embodiments may be described and/or claimed.

Frame interpolation via adaptive convolution and adaptive separable convolution

This disclosure describes implementations of a three-dimensional (3D) scene recovery system that reconstructs a 3D scene representation of a scene portrayed in a single digital image. For instance, the 3D scene recovery system trains and utilizes a 3D point cloud model to recover accurate intrinsic camera parameters from a depth map of the digital image. Additionally, the 3D point cloud model may include multiple neural networks that target specific intrinsic camera parameters. For example, the 3D point cloud model may include a depth 3D point cloud neural network that recovers the depth shift as well as include a focal length 3D point cloud neural network that recovers the camera focal length. Further, the 3D scene recovery system may utilize the recovered intrinsic camera parameters to transform the single digital image into an accurate and realistic 3D scene representation, such as a 3D point cloud.

Reconstructing three-dimensional scenes portrayed in digital images utilizing point cloud machine-learning models

Systems and methods are described for generating a three dimensional (3D) effect from a two dimensional (2D) image. The methods may include generating a depth map based on a 2D image, identifying a camera path, generating one or more extremal views based on the 2D image and the camera path, generating a global point cloud by inpainting occlusion gaps in the one or more extremal views, generating one or more intermediate views based on the global point cloud and the camera path, and combining the one or more extremal views and the one or more intermediate views to produce a 3D motion effect.

Growing community of inventors

Portland, OR, United States of America

Simon Niklaus