Adobe, Inc.

Google Inc.

Daniel Goldman

Elya Shechtman

Aliakbar Darabi

Ricardo Martin Brualla

Steven Maxwell Seitz

Andrew Ian Russell

Andrew Gerritt Huibers

Rohit Kumar Pandey

Jason Lawrence

Linjie Luo

Aseem Agarwala

Hailin Jin

Andrew Gerrit Huibers

Sofien Bouaziz

Supreeth Achar

Hugues Hoppe

David Edward Jacobs

Sujit Kuthirummal

Ohad I Fried

Jan Jachnik

Moustafa Meshry

Jen-Chan Jeff Chien

Sohrab Amirghodsi

Ke Li

Connelly Barnes

Simon Chen

Adam Finkelstein

Daniel Lischinski

Yoav HaCohen

Sameh Khamis

Matthew Alun Brown

Ian D Sachs

Noah Snavely

Harris Samuel Nover

Daniel Leventhal

Lynn Tsai

Lars Peter Johannes Hedman

Nima Khademi Kalantari

Aseem Agarwaia

Matthew Brown

Systems and methods are described for receiving a plurality of input images, a plurality of depth images, and a plurality of view parameters for generating a virtual view of a target subject. The systems and methods may generate a plurality of warped images based on the plurality of input images, the plurality of view parameters, and at least one of the plurality of depth images. In response to providing the plurality of depth images, the plurality of view parameters, and the plurality of warped images to a neural network, the systems and methods may receive, from the neural network, blending weights for assigning color to pixels of the virtual view of the target subject and may generate, based on the blending weights and the virtual view, a synthesized image according to the view parameters.

Neural blending for novel view synthesis

A method including capturing a first image of a real-world scene by a first camera sensitive to a first spectrum of light, the first camera having a first light source, capturing a second image of the real-world scene by a second camera sensitive to a second spectrum of light, the second camera having a second light source, identifying at least one feature in the first image, identifying, using a machine learning (ML) model, at least one feature in the second image that matches the at least one feature identified in the first image, mapping pixels in the first image and the second image to rays in a three-dimensional (3D) space based on the matched at least one feature, and calibrating the first camera and the second camera based on the mapping.

Cross spectral feature mapping for camera calibration

Systems and methods are described for utilizing an image processing system with at least one processing device to perform operations including receiving a plurality of input images of a user, generating a three-dimensional mesh proxy based on a first set of features extracted from the plurality of input images and a second set of features extracted from the plurality of input images. The method may further include generating a neural texture based on a three-dimensional mesh proxy and the plurality of input images, generating a representation of the user including at least a neural texture, and sampling at least one portion of the neural texture from the three-dimensional mesh proxy. In response to providing the at least one sampled portion to a neural renderer, the method may include receiving, from the neural renderer, a synthesized image of the user that is previously not captured by the image processing system.

Few-shot synthesis of talking heads

Systems and methods are described for generating a plurality of three-dimensional (3D) proxy geometries of an object, generating, based on the plurality of 3D proxy geometries, a plurality of neural textures of the object, the neural textures defining a plurality of different shapes and appearances representing the object, providing the plurality of neural textures to a neural renderer, receiving, from the neural renderer and based on the plurality of neural textures, a color image and an alpha mask representing an opacity of at least a portion of the object, and generating a composite image based on the pose, the color image, and the alpha mask.

Generative latent textured proxies for object category modeling

According to an aspect, a method for neural rerendering includes obtaining a three-dimensional (3D) model representing a scene of a physical space, where the 3D model is constructed from a collection of input images, rendering an image data buffer from the 3D model according to a viewpoint, where the image data buffer represents a reconstructed image from the 3D model, receiving, by a neural rerendering network, the image data buffer, receiving, by the neural rerendering network, an appearance code representing an appearance condition, and transforming, by the neural rerendering network, the image data buffer into a rerendered image with the viewpoint of the image data buffer and the appearance condition specified by the appearance code.

Neural rerendering from 3D models

Techniques of smoothing surface normals in a texture mapping application involve generating smoothed normals from the perspective of each camera using to capture images for texture mapping. Along these lines, a camera used to capture an image for texture mapping is situated at an orientation relative to the geometrical object onto which a texture mapping computer maps the texture image. The texture mapping computer places a filter window centered at a point on the geometrical object. The texture mapping computer then generates, as the smoothed normal at that point, an average normal over points in the filter window. The average normals thus computed for each camera are then used in the weights of the weighted average that is the image value at that point.

Smoothed normals from depth maps for normal-based texture blending

According to an aspect, a real-time active stereo system includes a capture system configured to capture stereo image data, where the image data includes a plurality of pairs of a reference image and a secondary image, and each pair of the plurality of pairs relates a different temporal window. The real-time active stereo system includes a depth sensing computing system including at least one processor and a non-transitory computer-readable medium having executable instructions that when executed by the at least one processor are configured to execute a local stereo reconstruction algorithm configured to compute spacetime descriptors from the plurality of pairs of the stereo image data and generate depth maps based on the spacetime descriptors.

Real-time spacetime stereo using spacetime descriptors

An example telepresence terminal includes a display, an image sensor, an infrared emitter, and an infrared depth sensor. The terminal may determine image data using visible light emitted by the infrared emitter and captured by the image sensor and determine depth data using infrared light captured by the infrared depth sensor. The terminal may also communicate the depth data and the image data to a remote telepresence terminal and receive remote image data and remote depth data. The terminal may also generate a first display image using the lenticular display based on the remote image data that is viewable from a first viewing location and generate a second display image using the lenticular display based on the remote image data and the remote depth data that is viewable from a second viewing location.

Three-dimensional telepresence system

An example telepresence terminal includes a lenticular display, an image sensor, an infrared emitter, and an infrared depth sensor. The terminal may determine image data using visible light emitted by the infrared emitter and captured by the image sensor and determine depth data using infrared light captured by the infrared depth sensor. The terminal may also communicate the depth data and the image data to a remote telepresence terminal and receive remote image data and remote depth data. The terminal may also generate a first display image using the lenticular display based on the remote image data that is viewable from a first viewing location and generate a second display image using the lenticular display based on the remote image data and the remote depth data that is viewable from a second viewing location.

Image distractor detection and processing techniques are described. In one or more implementations, a digital medium environment is configured for image distractor detection that includes detecting one or more locations within the image automatically and without user intervention by the one or more computing devices that include one or more distractors that are likely to be considered by a user as distracting from content within the image. The detection includes forming a plurality of segments from the image by the one or more computing devices and calculating a score for each of the plurality of segments that is indicative of a relative likelihood that a respective said segment is considered a distractor within the image. The calculation is performed using a distractor model trained using machine learning as applied to a plurality images having ground truth distractor locations.

Image distractor detection and processing

Variable patch shape synthesis techniques are described. In one or more implementations, a plurality of patches are computed from one or more images, at least one of the plurality of patches having a different shape than another one of the plurality of patches. The shapes define an area to be considered for use in a patch synthesis technique. The patch synthesis technique is performed to edit an image using the computed plurality of patches having the different shapes.

Variable patch shape synthesis

One embodiment involves receiving a fine mesh as input, the fine mesh representing a 3-Dimensional (3D) model and comprising fine mesh polygons. The embodiment further involves identifying, based on the fine mesh, near-planar regions represented by a coarse mesh of coarse mesh polygons, at least one of the near-planar regions corresponding to a plurality of the coarse mesh polygons. The embodiment further involves determining a deformation to deform the coarse mesh based on comparing normals between adjacent coarse mesh polygons. The deformation may involve reducing a first angle between coarse mesh polygons adjacent to one another in a same near-planar region. The deformation may additionally or alternatively involve increasing an angle between coarse mesh polygons adjacent to one another in different near-planar regions. The fine mesh can be deformed using the determined deformation.

3D model enhancement

Techniques for focal length warping are described. Focal length warping, for instance, may provide an automated approach for correcting distortion in an input image to improve its perceptual quality. In at least some implementations, a focal length of a camera lens used to capture an image and an estimated camera distance are utilized to three-dimensionally reproject and warp the image to generate an adjusted image simulating a new focal length and a new camera distance. Implementations of focal length warping may estimate a camera distance based on facial features in an image.

Focal length warping

Embodiments of the present invention provide systems, methods, and computer storage media directed at image synthesis utilizing an active mask. In one embodiment, input is received that identifies a target region within an image that is to be synthesized. A patch synthesis technique can then be performed to synthesize the target region based on portions of a source region that are identified by the patch synthesis technique. In embodiments, the patch synthesis technique includes, for at least one iteration, generating an active mask that indicates one or more portions of the target region as inactive. This active mask can be utilized by at least one process of the patch synthesis technique to ignore the one or more portions indicated as inactive by the active mask for the at least one iteration of the patch synthesis technique. Other embodiments may be described and/or claimed.

Growing community of inventors

Seattle, WA, United States of America

Daniel Goldman