Adobe, Inc.

Universitè Laval

Nathan Aaron Carr

Radomir Mech

Kalyan Krishna Sunkavalli

Xin Sun

Gavin S P Miller

Elya Shechtman

Pushkar Prakash Joshi

Hailin Jin

Gavin Stuart Peter Miller

Matheus Gadelha

Jonathan Worthen Brandt

Gregg Darryl Wilensky

Sunil Sharadchandra Hadap

Michal Lukac

Scott David Cohen

Zeke Koch

Zhili Chen

Walter Wei-Tuh Chang

Vojtech Krs

Vineet Batra

Ankit Phogat

Byungmoon Kim

Stephen N Schiller

Duygu Ceylan

Aravind Krishnaswamy

Tamy Boubekeur

Daichi Ito

Sumit Dhingra

Mathieu Garon

Viswanathan Swaminathan

Jingwan Lu

Oliver Wang

Linjie Luo

Sohrab Amirghodsi

Richard Zhang

Duygu Ceylan Aksit

Sarah Kong

Shabnam Ghadar

Mehmet Yumer

Ratheesh Kalarot

Niloy Jyoti Mitra

Stefano Petrangeli

Alan Lee Erickson

Zhixin Shu

Cameron Smith

Yuyan Song

Wei-An Lin

Milos Hasan

Giorgio Gori

Bradee Rae Evans

Jun-Yan Zhu

Kevin Karsch

Gwendal Simon

James L Andrews

Jared M Johnson

Yannick Hold-Geoffroy

Yukie Takahashi

Yannick Hold-Geoffrey

Hao Qin

Tim Christopher Reiner

Naoki Sasaki

Manish Kumar

Michael J Orts

Meredith L Stotzner

Heather M Dolan

Wojciech Jarosz

Zhaowen Wang

Praveen Kumar Dhanuka

Jimei Yang

Menglei Chai

Jean-Francois Lalonde

Peter F Falco, Jr

Siddhartha Chaudhuri

Grayson Squier Lang

Michal Lukác

Nikolai Svakhin

Eric Vinchon

Ashish Jindal

Michael O Clifton

Qi-xing Huang

Alexandr Kuznetsov

Theo Thonat

Fatemeh Abbasinejad

Francois Beaune

I-Chao Shen

Bochang Moon

Tobias O Martin

Lu Liu

Adam G Kirk

Julio Marco Murria

Siying Liu

Weilun Sun

Tsukasa Fukusato

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images. The disclosed systems further use three-dimensional representations of two-dimensional images to customize focal points for the two-dimensional images.

Generating shadows for objects in two-dimensional images utilizing a plurality of shadow maps

Generating shadows for placed objects in depth estimated scenes of two-dimensional images

Methods, systems, and non-transitory computer readable storage media are disclosed for generating three-dimensional meshes representing two-dimensional images for editing the two-dimensional images. The disclosed system utilizes a first neural network to determine density values of pixels of a two-dimensional image based on estimated disparity. The disclosed system samples points in the two-dimensional image according to the density values and generates a tessellation based on the sampled points. The disclosed system utilizes a second neural network to estimate camera parameters and modify the three-dimensional mesh based on the estimated camera parameters of the pixels of the two-dimensional image. In one or more additional embodiments, the disclosed system generates a three-dimensional mesh to modify a two-dimensional image according to a displacement input. Specifically, the disclosed system maps the three-dimensional mesh to the two-dimensional image, modifies the three-dimensional mesh in response to a displacement input, and updates the two-dimensional image.

Modifying two-dimensional images utilizing three-dimensional meshes of the two-dimensional images

An improved system architecture uses a Generative Adversarial Network (GAN) including a specialized generator neural network to generate multiple resolution output images. The system produces a latent space representation of an input image. The system generates a first output image at a first resolution by providing the latent space representation of the input image as input to a generator neural network comprising an input layer, an output layer, and a plurality of intermediate layers and taking the first output image from an intermediate layer, of the plurality of intermediate layers of the generator neural network. The system generates a second output image at a second resolution different from the first resolution by providing the latent space representation of the input image as input to the generator neural network and taking the second output image from the output layer of the generator neural network.

Multi-scale output techniques for generative adversarial networks

Modifying two-dimensional images utilizing iterative three-dimensional meshes of the two-dimensional images

Modifying two-dimensional images utilizing segmented three-dimensional object meshes of the two-dimensional images

In implementations of systems for visualizing vector graphics in three-dimensional scenes, a computing device implements a projection system to receive input data describing a digital image depicting a three-dimensional scene and a vector graphic to be projected into the three-dimensional scene. The projection system generates a depth image by estimating disparity values for pixels of the digital image. A three-dimensional mesh is computed that approximates the three-dimensional scene based on the depth image. The projection system projects the vector graphic onto the digital image by transforming the vector graphic based on the three-dimensional mesh.

Visualizing vector graphics in three-dimensional scenes

Embodiments are disclosed for performing 3-D vectorization. The method includes obtaining a three-dimensional rendered image and a camera position. The method further includes obtaining a triangle mesh representing the three-dimensional rendered image. The method further involves creating a reduced triangle mesh by removing one or more triangles from the triangle mesh. The method further involves subdividing each triangle of the reduced triangle mesh into one or more subdivided triangles. The method further involves performing a mapping of each pixel of the three-dimensional rendered image to the reduced triangle mesh. The method further involves assigning a color value to each vertex of the reduced triangle mesh. The method further involves sorting each triangle of the reduced triangle mesh using a depth value of each triangle. The method further involves generating a two-dimensional triangle mesh using the sorted triangles of the reduced triangle mesh.

Resolution independent 3-D vectorization for graphic designs

Aspects and features of the present disclosure provide a direct ray tracing operator with a low memory footprint for surfaces enriched with displacement maps. A graphics editing application can be used to manipulate displayed representations of a 3D object that include surfaces with displacement textures. The application creates an independent map of a displaced surface. The application ray-traces bounding volumes on the fly and uses the intersection of a query ray with a bounding volume to produce rendering information for a displaced surface. The rendering information can be used to generate displaced surfaces for various base surfaces without significant re-computation so that updated images can be rendered quickly, in real time or near real time.

Displacement-centric acceleration for ray tracing

An improved system architecture uses a pipeline including a Generative Adversarial Network (GAN) including a generator neural network and a discriminator neural network to generate an image. An input image in a first domain and information about a target domain are obtained. The domains correspond to image styles. An initial latent space representation of the input image is produced by encoding the input image. An initial output image is generated by processing the initial latent space representation with the generator neural network. Using the discriminator neural network, a score is computed indicating whether the initial output image is in the target domain. A loss is computed based on the computed score. The loss is minimized to compute an updated latent space representation. The updated latent space representation is processed with the generator neural network to generate an output image in the target domain.

Techniques for domain to domain projection using a generative model

The present disclosure is directed toward systems, methods, and non-transitory computer readable media for generating a modified digital image by identifying patch matches within a digital image utilizing a Gaussian mixture model. For example, the systems described herein can identify sample patches and corresponding matching portions within a digital image. The systems can also identify transformations between the sample patches and the corresponding matching portions. Based on the transformations, the systems can generate a Gaussian mixture model, and the systems can modify a digital image by replacing a target region with target matching portions identified in accordance with the Gaussian mixture model.

Performing patch matching guided by a transformation gaussian mixture model

Generative shape creation and editing is leveraged in a digital medium environment. An object editor system represents a set of training shapes as sets of visual elements known as 'handles,' and converts sets of handles into signed distance field (SDF) representations. A handle processor model is then trained using the SDF representations to enable the handle processor model to generate new shapes that reflect salient visual features of the training shapes. The trained handle processor model, for instance, generates new sets of handles based on salient visual features learned from the training handle set. Thus, utilizing the described techniques, accurate characterizations of a set of shapes can be learned and used to generate new shapes. Further, generated shapes can be edited and transformed in different ways.

Generative shape creation and editing

The present disclosure relates to systems, non-transitory computer-readable media, and methods that utilize hemispherical clamping for importance sampling of an image-based light (IBL) to generate a digital image of a virtual environment. For example, the disclosed systems identify a hemispherical portion of an IBL image that corresponds to a reflective surface location on a virtual object. The disclosed systems can then clamp the IBL image using one or more importance sampling algorithms to exclude portions of the IBL image outside of the hemispherical portion that do not contribute direct lighting onto the reflective surface location. The disclosed systems can further utilize the one or more importance sampling algorithms to efficiently sample a ray direction between the reflective surface location and the hemispherical portion of the IBL image. In certain embodiments, the disclosed systems use the sampled ray direction to generate a digital image rendering portraying the virtual object.

Utilizing hemispherical clamping for importance sampling of image-based light to render a virtual environment

Techniques and systems are provided for generating a video from texture images, and for reconstructing the texture images from the video. For example, a texture image can be divided into a number of tiles, and the number of tiles can be sorted into a sequence of ordered tiles. The sequence of ordered tiles can be provided to a video coder for generating a coded video. The number of tiles can be encoded based on the sequence of ordered tiles. The encoded video including the encoded sequence of ordered tiles can be decoded. At least a portion of the decoded video can include the number of tiles sorted into a sequence of ordered tiles. A data file associated with at least the portion of the decoded video can be used to reconstruct the texture image using the tiles.

Generation of a sequence of textures for video delivery

Image modification using detected symmetry is described. In example implementations, an image modification module detects multiple local symmetries in an original image by discovering repeated correspondences that are each related by a transformation. The transformation can include a translation, a rotation, a reflection, a scaling, or a combination thereof. Each repeated correspondence includes three patches that are similar to one another and are respectively defined by three pixels of the original image. The image modification module generates a global symmetry of the original image by analyzing an applicability to the multiple local symmetries of multiple candidate homographies contributed by the multiple local symmetries. The image modification module associates individual pixels of the original image with a global symmetry indicator to produce a global symmetry association map. The image modification module produces a manipulated image by manipulating the original image under global symmetry constraints imposed by the global symmetry association map.

Image modification using detected symmetry

Growing community of inventors

San Jose, CA, United States of America

Nathan Aaron Carr