Adobe, Inc.

Geoffrey Oxholm

Elya Shechtman

Oliver Wang

John Nelson

Matthew David Fisher

Jonathan Eisenmann

Wenqi Xian

Ramiz Sheikh

Zhili Chen

Duygu Ceylan Aksit

Seth John Walker

Geoffrey Charles Dowd

John Philip Collomosse

Bryan Russell

Michal Lukac

Xiaoyang Li

Xin Wang

Maksym Andriushchenko

Pranjali Kokare

One or more processing devices access a scene depicting a reference object that includes an annotation identifying a target region to be modified in one or more video frames. The one or more processing devices determine that a target pixel corresponds to a sub-region within the target region that includes hallucinated content. The one or more processing devices determine gradient constraints using gradient values of neighboring pixels in the hallucinated content, the neighboring pixels being adjacent to the target pixel and corresponding to four cardinal directions. The one or more processing devices update color data of the target pixel subject to the determined gradient constraints.

Corrective lighting for video inpainting

The present disclosure relates to systems, methods, and non-transitory computer readable media that utilize a deep visual fingerprinting model with parameters learned from robust contrastive learning to identify matching digital images and image provenance information. For example, the disclosed systems utilize an efficient learning procedure that leverages training on bounded adversarial examples to more accurately identify digital images (including adversarial images) with a small computational overhead. To illustrate, the disclosed systems utilize a first objective function that iteratively identifies augmentations to increase contrastive loss. Moreover, the disclosed systems utilize a second objective function that iteratively learns parameters of a deep visual fingerprinting model to reduce the contrastive loss. With these learned parameters, the disclosed systems utilize the deep visual fingerprinting model to generate visual fingerprints for digital images, retrieve and match digital images, and provide digital image provenance information.

Adversarially robust visual fingerprinting and image provenance models

Certain aspects involve video inpainting in which content is propagated from a user-provided reference video frame to other video frames depicting a scene. One example method includes one or more processing devices that performs operations that include accessing a scene depicting a reference object that includes an annotation identifying a target region to be modified in one or more video frames. The operations also includes computing a target motion of a target pixel that is subject to a motion constraint. The motion constraint is based on a three-dimensional model of the reference object. Further, operations include determining color data of the target pixel to correspond to the target motion. The color data includes a color value and a gradient. Operations also include determining gradient constraints using gradient values of neighbor pixels. Additionally, the processing devices updates the color data of the target pixel subject to the gradient constraints.

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing a critical edge detection neural network and a geometric model to determine camera parameters from a single digital image. In particular, in one or more embodiments, the disclosed systems can train and utilize a critical edge detection neural network to generate a vanishing edge map indicating vanishing lines from the digital image. The system can then utilize the vanishing edge map to more accurately and efficiently determine camera parameters by applying a geometric model to the vanishing edge map. Further, the system can generate ground truth vanishing line data from a set of training digital images for training the critical edge detection neural network.

Determining camera parameters from a single digital image

Certain aspects involve video inpainting in which content is propagated from a user-provided reference frame to other video frames depicting a scene. For example, a computing system accesses a set of video frames with annotations identifying a target region to be modified. The computing system determines a motion of the target region's boundary across the set of video frames, and also interpolates pixel motion within the target region across the set of video frames. The computing system also inserts, responsive to user input, a reference frame into the set of video frames. The reference frame can include reference color data from a user-specified modification to the target region. The computing system can use the reference color data and the interpolated motion to update color data in the target region across set of video frames.

Video inpainting via machine-learning models with motion constraints

Utilizing a critical edge detection neural network and a geometric model to determine camera parameters from a single digital image

Certain aspects involve video inpainting via confidence-weighted motion estimation. For instance, a video editor accesses video content having a target region to be modified in one or more video frames. The video editor computes a motion for a boundary of the target region. The video editor interpolates, from the boundary motion, a target motion of a target pixel within the target region. In the interpolation, confidence values assigned to boundary pixels control how the motion of these pixels contributes to the interpolated target motion. A confidence value is computed based on a difference between forward and reverse motion with respect to a particular boundary pixel, a texture in a region that includes the particular boundary pixel, or a combination thereof. The video editor modifies the target region in the video by updating color data of the target pixel to correspond to the target motion interpolated from the boundary motion.

Video inpainting via confidence-weighted motion estimation

Video inpainting via user-provided reference frame

A computing system transforms an input image into a stylized output image by applying first and second style features from a style exemplar. The input image is provided to a multimodal style-transfer network having a low-resolution-based stylization subnet and a high-resolution stylization subnet. The low-resolution-based stylization subnet is trained with low-resolution style exemplars to apply the first style feature. The high-resolution stylization subnet is trained with high-resolution style exemplars to apply the second style feature. The low-resolution-based stylization subnet generates an intermediate image by applying the first style feature from a low-resolution version of the style exemplar to first image data obtained from the input image. Second image data from the intermediate image is provided to the high-resolution stylization subnet. The high-resolution stylization subnet generates the stylized output image by applying the second style feature from a high-resolution version of the style exemplar to the second image data.

Multimodal style-transfer network for applying style features from multi-resolution style exemplars to input images

Provided are video processing architectures and techniques configured to generate looping video. The video processing architectures and techniques automatically produce a looping video from a fixed-length video clip. Embodiments of the video processing architectures and techniques determine a lower-resolution version of the fixed-length video clip, and detect a presence of edges within image frames in the lower-resolution version. A pair of image frames having similar edges is identified as a pair of candidates for a transition point (i.e., a start frame and an end frame) at which the looping video can repeat. Using start and end frames having similar edges mitigates teleporting of objects displayed in the looping video. In some cases, teleporting during repeating is eliminated.

Video processing architectures which provide looping video

Embodiments disclosed herein provide systems, methods, and computer storage media for automatically aligning a 3D camera with a 2D background image. An automated image analysis can be performed on the 2D background image, and a classifier can predict whether the automated image analysis is accurate within a selected confidence level. As such, a feature can be enabled that allows a user to automatically align the 3D camera with the 2D background image. For example, where the automated analysis detects a horizon and one or more vanishing points from the background image, the 3D camera can be automatically transformed to align with the detected horizon and to point at a detected horizon-located vanishing point. In some embodiments, 3D objects in a 3D scene can be pivoted and the 3D camera dollied forward or backwards to reduce changes to the framing of the 3D composition resulting from the 3D camera transformation.

Automatic 3D camera alignment and object arrangment to match a 2D background image

Certain embodiments involve switching to a particular video loop based on a user's focus while displaying video content to a user. For example, a video playback system identifies a current frame of the video content in a current region of interest being presented to a user on a presentation device. The video playback system also identifies an updated region of interest in the video content. The video playback system determines that a start frame and an end frame of a video portion within the region of interest have a threshold similarity. The video playback system selects, based on the threshold similarity, the video portion as a loop to be displayed to the user in the updated region of interest. The video playback system causes the presentation device to display the video portion as the loop.

Focus-based video loop switching

Systems and methods disclosed herein provide drawing assistance to a user that improves the user's ability to draw accurate perspective drawings. In general, a drawing assistance system provides a homography that annotates guidelines over an image and connects lined features of the image to a corresponding vanishing point. More specifically, the homography provides a perspective grid to the user that assists the user in adding drawing strokes to a drawing using the proper perspective. In addition, the drawing assistance system can adjust a user's drawing stroke to properly align with a vanishing point. Further, in some embodiments, the drawing assistance system allows a user to add an asset into a drawing such that the asset is displayed in the proper perspective, even after a user moves the asset within a drawing.

Growing community of inventors

Albany, CA, United States of America

Geoffrey Oxholm