Adobe, Inc.

Disney Enterprises, Inc.

Eidgenossische Technische Hochschule Zurich

Eth Zurich

Federico Perazzi

Zhe Lin

Alexander Sorkine Hornung

Jianming Zhang

Oliver Wang

Yilin Wang

Michael Gharbi

Shikun Liu

Kalyan Krishna Sunkavalli

Jingwan Lu

Christopher Richard Schroers

He Zhang

Henning Zimmer

Yael Pritch

Fabian David Caba Heilbron

Zhihao Xia

Ping Hu

Philipp Krähenbühl

Scott Frazier Watson

Matthew David Fisher

Baldo Antonio Faieta

Mingyang Ling

Jonathan Eisenmann

Paul A Beardsley

Brian McWilliams

Nico Becherer

Andreas Breitenmoser

Markus Woodson

Brett Michael Butterfield

Mario Botsch

Peter Kaufmann

Javier Alonso Mora

Jason Kuen

Krishna Bhargava Mullia Lakshminarayana

Yifan Wang

Nicolas Märki

Mustafa Isik

Caner Hazirbas

Nikhil Kalra

The present disclosure relates to an image retouching system that automatically retouches digital images by accurately correcting face imperfections such as skin blemishes and redness. For instance, the image retouching system automatically retouches a digital image through separating digital images into multiple frequency layers, utilizing a separate corresponding neural network to apply frequency-specific corrections at various frequency layers, and combining the retouched frequency layers into a retouched digital image. As described herein, the image retouching system efficiently utilizes different neural networks to target and correct skin features specific to each frequency layer.

Retouching digital images utilizing layer specific deep-learning neural networks

A plurality of pixel-based sampling points are identified within an image, wherein sampling points of a pixel are distributed within the pixel. For individual sampling points of individual pixels, a corresponding radiance vector is estimated. A radiance vector includes one or more radiance values characterizing light received at a sampling point. A first machine learning module generates, for each pixel, a corresponding intermediate radiance feature vector, based on the radiance vectors associated with the sampling points within that pixel. A second machine learning module generates, for each pixel, a corresponding final radiance feature vector, based on an intermediate radiance feature vector for that pixel, and one or more other intermediate radiance feature vectors for one or more other pixels neighboring that pixel. One or more kernels are generated, based on the final radiance feature vectors, and applied to corresponding pixels of the image, to generate a lower noise image.

Denoising images rendered using Monte Carlo renderings

The disclosure describes one or more implementations of a neural network architecture pruning system that automatically and progressively prunes neural networks. For instance, the neural network architecture pruning system can automatically reduce the size of an untrained or previously-trained neural network without reducing the accuracy of the neural network. For example, the neural network architecture pruning system jointly trains portions of a neural network while progressively pruning redundant subsets of the neural network at each training iteration. In many instances, the neural network architecture pruning system increases the accuracy of the neural network by progressively removing excess or redundant portions (e.g., channels or layers) of the neural network. Further, by removing portions of a neural network, the neural network architecture pruning system can increase the efficiency of the neural network.

Generating and utilizing pruned neural networks

A Video Semantic Segmentation System (VSSS) is disclosed that performs accurate and fast semantic segmentation of videos using a set of temporally distributed neural networks. The VSSS receives as input a video signal comprising a contiguous sequence of temporally-related video frames. The VSSS extracts features from the video frames in the contiguous sequence and based upon the extracted features, selects, from a set of labels, a label to be associated with each pixel of each video frame in the video signal. In certain embodiments, a set of multiple neural networks are used to extract the features to be used for video segmentation and the extraction of features is distributed among the multiple neural networks in the set. A strong feature representation representing the entirety of the features is produced for each video frame in the sequence of video frames by aggregating the output features extracted by the multiple neural networks.

Temporally distributed neural networks for video semantic segmentation

Certain embodiments involve techniques for efficiently estimating denoising kernels for generating denoised images. For instance, a neural network receives a noisy reference image to denoise. The neural network uses a kernel dictionary of base kernels and generates a coefficient vector for each pixel in the reference image such that the coefficient vector includes a coefficient value for each base kernel in the kernel dictionary, where the base kernels are combined to generate a denoising kernel and each coefficient value indicates a contribution of a given base kernel to a denoising kernel. The neural network calculates the denoising kernel for a given pixel by applying the coefficient vector for that pixel to the kernel dictionary. The neural network applies each denoising kernel to the respective pixel to generate a denoised output image.

Kernel prediction with kernel dictionary in image denoising

The present disclosure relates to shaping the architecture of a neural network. For example, the disclosed systems can provide a neural network shaping mechanism for at least one sampling layer of a neural network. The neural network shaping mechanism can include a learnable scaling factor between a sampling rate of the at least one sampling layer and an additional sampling function. The disclosed systems can learn the scaling factor based on a dataset while jointly learning the network weights of the neural network. Based on the learned scaling factor, the disclosed systems can shape the architecture of the neural network by modifying the sampling rate of the at least one sampling layer.

Shaping a neural network architecture utilizing learnable sampling layers

Neural network architecture pruning

The present disclosure relates to utilizing a neural network having a two-stream encoder architecture to accurately generate composite digital images that realistically portray a foreground object from one digital image against a scene from another digital image. For example, the disclosed systems can utilize a foreground encoder of the neural network to identify features from a foreground image and further utilize a background encoder to identify features from a background image. The disclosed systems can then utilize a decoder to fuse the features together and generate a composite digital image. The disclosed systems can train the neural network utilizing an easy-to-hard data augmentation scheme implemented via self-teaching. The disclosed systems can further incorporate the neural network within an end-to-end framework for automation of the image composition process.

Utilizing a two-stream encoder neural network to generate composite digital images

Retouching digital images utilizing separate deep-learning neural networks

In implementations of determining video cuts in video clips, a video cut detection system can receive a video clip that includes a sequence of digital video frames that depict one or more scenes. The video cut detection system can determine scene characteristics for the digital video frames. The video cut detection system can determine, from the scene characteristics, a probability of a video cut between two adjacent digital video frames having a boundary between the two adjacent digital video frames that is centered in the sequence of digital video frames. The video cut detection system can then compare the probability of the video cut to a cut threshold to determine whether the video cut exists between the two adjacent digital video frames.

Determining video cuts in video clips

Utilizing a neural network having a two-stream encoder architecture to generate composite digital images

Systems and methods are described for object detection within a digital image using a hierarchical softmax function. The method may include applying a first softmax function of a softmax hierarchy on a digital image based on a first set of object classes that are children of a root node of a class hierarchy, then apply a second (and subsequent) softmax functions to the digital image based on a second (and subsequent) set of object classes, where the second (and subsequent) object classes are children nodes of an object class from the first (or parent) object classes. The methods may then include generating an object recognition output using a convolutional neural network (CNN) based at least in part on applying the first and second (and subsequent) softmax functions. In some cases, the hierarchical softmax function is the loss function for the CNN.

Robust training of large-scale object detectors with a noisy dataset

According to one implementation, an image processing system includes a computing platform having a hardware processor and a system memory storing a software code including a convolutional neural network (CNN) trained using one or more semantic map(s). The hardware processor executes the software code to receive an original image including multiple object images each identified with one of multiple object classes, and to generate replications of the original image, each replication corresponding respectively to one of the object classes. The hardware processor further executes the software code to, for each replication, selectively modify one or more object image(s) identified with the object class corresponding to the replication, using the CNN, to produce partially modified images each corresponding respectively to an object class, and to merge the partially modified images, using the CNN, to generate a modified image corresponding to the original image.

Image processing using a convolutional neural network

Growing community of inventors

San Francisco, CA, United States of America

Federico Perazzi