Adobe, Inc.

Microsoft Technology Licensing, LLC

Silicon Graphics, Incorporated

Graphics Properties Holdings, Inc.

Nvidia Corporation

Radomir Mech

Zhe Lin

Xiaohui Shen

Nathan Aaron Carr

Jianming Zhang

Gavin Stuart Peter Miller

Matheus Gadelha

Vojtech Krs

Tamy Boubekeur

Giorgio Gori

Brian Lynn Price

Jonathan Worthen Brandt

Duygu Ceylan Aksit

Hailin Jin

Scott David Cohen

Kalyan Krishna Sunkavalli

Chen Fang

Zeke Koch

Mehmet Yumer

Walter Wei-Tuh Chang

Milos Hasan

Jen-Chan Jeff Chien

Jingwan Lu

Xin Lu

Anant Gilra

Gavin S P Miller

Peng Wang

Zijun Wei

Yufei Wang

Lu Zhang

Liang Shi

Shu Kong

Jian Ren

Stephen J DiVerdi

Paul John Asente

Aravind Krishnaswamy

Zahid S Hussain

Vladimir Kim

Emiliano Gambaretto

Kevin Wampler

Jonathan Eisenmann

Daichi Ito

Noam Aigerman

Siddhartha Chaudhuri

Gianpaolo Tommasi

Yannick Hold-Geoffroy

Mathieu Gaillard

Alannah Oleson

Jose Echevarria

Tim Christopher Reiner

Fan Bao

Philip Nemec

Ning Xu

Byungmoon Kim

Jose Ignacio Echevarria Vallespi

Niloy Jyoti Mitra

Angus Dorbie

Daniel Elliot Wexler

Larry I Gritz

Eric Enderton

Peter F Falco, Jr

Yi-Ting Yeh

Nikolai Svakhin

Eric Vinchon

Qi-xing Huang

Biao Jia

Paul Augusto Guerrero

Thibault Groueix

Minhyuk Sung

Kevin Armin Samii

Yair Kurzion

Minghua Liu

Ning Yu

Craig Kaplan

Zorana Gee

Jerry O Talton, Iii

John F Schlag

Takashi Ijiri

Jane Little E

Jason L Freund

Marissa Ramirez De Chanlatte

Weiyue Wang

Shanghang Zhang

Reza Adhitya Saputra

Tsukasa Fukusato

Haibin Huang

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify two-dimensional images via scene-based editing using three-dimensional representations of the two-dimensional images. For instance, in one or more embodiments, the disclosed systems utilize three-dimensional representations of two-dimensional images to generate and modify shadows in the two-dimensional images according to various shadow maps. Additionally, the disclosed systems utilize three-dimensional representations of two-dimensional images to modify humans in the two-dimensional images. The disclosed systems also utilize three-dimensional representations of two-dimensional images to provide scene scale estimation via scale fields of the two-dimensional images. In some embodiments, the disclosed systems utilizes three-dimensional representations of two-dimensional images to generate and visualize 3D planar surfaces for modifying objects in two-dimensional images. The disclosed systems further use three-dimensional representations of two-dimensional images to customize focal points for the two-dimensional images.

Generating shadows for objects in two-dimensional images utilizing a plurality of shadow maps

Generating shadows for placed objects in depth estimated scenes of two-dimensional images

Methods, systems, and non-transitory computer readable storage media are disclosed for generating three-dimensional meshes representing two-dimensional images for editing the two-dimensional images. The disclosed system utilizes a first neural network to determine density values of pixels of a two-dimensional image based on estimated disparity. The disclosed system samples points in the two-dimensional image according to the density values and generates a tessellation based on the sampled points. The disclosed system utilizes a second neural network to estimate camera parameters and modify the three-dimensional mesh based on the estimated camera parameters of the pixels of the two-dimensional image. In one or more additional embodiments, the disclosed system generates a three-dimensional mesh to modify a two-dimensional image according to a displacement input. Specifically, the disclosed system maps the three-dimensional mesh to the two-dimensional image, modifies the three-dimensional mesh in response to a displacement input, and updates the two-dimensional image.

Modifying two-dimensional images utilizing three-dimensional meshes of the two-dimensional images

Generating adaptive three-dimensional meshes of two-dimensional images

Modifying two-dimensional images utilizing iterative three-dimensional meshes of the two-dimensional images

Modifying two-dimensional images utilizing segmented three-dimensional object meshes of the two-dimensional images

Systems and methods provide editing operations in a smart editing system that may generate a focal point within a mask of an object for each frame of a video segment and perform editing effects on the frames of the video segment to quickly provide users with natural video editing effects. An eye-gaze network may produce a hotspot map of predicted focal points in a video frame. These predicted focal points may then be used by a gaze-to-mask network to determine objects in the image and generate an object mask for each of the detected objects. This process may then be repeated to effectively track the trajectory of objects and object focal points in videos. Based on the determined trajectory of an object in a video clip and editing parameters, the editing engine may produce editing effects relative to an object for the video clip.

System for automatic object mask and hotspot tracking

The present disclosure relates to using end-to-end differentiable pipeline for optimizing parameters of a base procedural material to generate a procedural material corresponding to a target physical material. For example, the disclosed systems can receive a digital image of a target physical material. In response, the disclosed systems can retrieve a differentiable procedural material for use as a base procedural material in response. The disclosed systems can compare a digital image of the base procedural material with the digital image of the target physical material using a loss function, such as a style loss function that compares visual appearance. Based on the determined loss, the disclosed systems can modify the parameters of the base procedural material to determine procedural material parameters for the target physical material. The disclosed systems can generate a procedural material corresponding to the base procedural material using the determined procedural material parameters.

Generating differentiable procedural materials

Procedural model digital content editing techniques are described that overcome the limitations of conventional techniques to make procedural models available for interaction by a wide range of users without requiring specialized knowledge and do so without 'breaking' the underlying model. In the techniques described herein, an inverse procedural model system receives a user input that specifies an edit to digital content generated by a procedural model. Input parameters from these candidate input parameters are selected by the system which cause the digital content generated by the procedural model to incorporate the edit.

Digital content editing using a procedural model

The present disclosure relates to systems, non-transitory computer-readable media, and methods that tune a 3D-object-reconstruction-machine-learning model to reconstruct 3D models of objects from real images using real images as training data. For instance, the disclosed systems can determine a depth map for a real two-dimensional (2D) image and then reconstruct a 3D model of a digital object in the real 2D image based on the depth map. By using a depth map for a real 2D image, the disclosed systems can generate reconstructed 3D models that better conform to the shape of digital objects in real images than existing systems and use such reconstructed 3D models to generate more realistic looking visual effects (e.g., shadows, relighting).

Reconstructing three-dimensional models of objects from real images based on depth information

Aspects of a system and method for procedural media generation include generating a sequence of operator types using a node generation network; generating a sequence of operator parameters for each operator type of the sequence of operator types using a parameter generation network; generating a sequence of directed edges based on the sequence of operator types using an edge generation network; combining the sequence of operator types, the sequence of operator parameters, and the sequence of directed edges to obtain a procedural media generator, wherein each node of the procedural media generator comprises an operator that includes an operator type from the sequence of operator types, a corresponding sequence of operator parameters, and an input connection or an output connection from the sequence of directed edges that connects the node to another node of the procedural media generator; and generating a media asset using the procedural media generator.

Procedural media generation

Generative shape creation and editing is leveraged in a digital medium environment. An object editor system represents a set of training shapes as sets of visual elements known as 'handles,' and converts sets of handles into signed distance field (SDF) representations. A handle processor model is then trained using the SDF representations to enable the handle processor model to generate new shapes that reflect salient visual features of the training shapes. The trained handle processor model, for instance, generates new sets of handles based on salient visual features learned from the training handle set. Thus, utilizing the described techniques, accurate characterizations of a set of shapes can be learned and used to generate new shapes. Further, generated shapes can be edited and transformed in different ways.

Generative shape creation and editing

Embodiments of the present invention are directed towards intuitive editing of three-dimensional models. In embodiments, salient geometric features associated with a three-dimensional model defining an object are identified. Thereafter, feature attributes associated with the salient geometric features are identified. A feature set including a plurality of salient geometric features related to one another is generated based on the determined feature attributes (e.g., properties, relationships, distances). An editing handle can then be generated and displayed for the feature set enabling each of the salient geometric features within the feature set to be edited in accordance with a manipulation of the editing handle. The editing handle can be displayed in association with one of the salient geometric features of the feature set.

Intuitive editing of three-dimensional models

Embodiments of the present invention are directed to facilitating region of interest preservation. In accordance with some embodiments of the present invention, a region of interest preservation score using adaptive margins is determined. The region of interest preservation score indicates an extent to which at least one region of interest is preserved in a candidate image crop associated with an image. A region of interest positioning score is determined that indicates an extent to which a position of the at least one region of interest is preserved in the candidate image crop associated with the image. The region of interest preservation score and/or the preserving score are used to select a set of one or more candidate image crops as image crop suggestions.

Preserving regions of interest in automatic image cropping

Growing community of inventors

Mountain View, CA, United States of America

Radomir Mech