Disney Enterprises, Inc.

Eth Zurich (eidgenossische Technische Hochschule Zurich)

Eth Zurich

Beijing Yojaja Software Technology Development Co., Ltd.

Pixar

Lucasfilm Entertainment Company Ltd.

Eth Zurich (eidgenssische Technische Hochschule Zurich)

Christopher Richard Schroers

Abdelaziz Djelouah

Scott C Labrozzi

Yuanyi Xue

Romann Matthew Weber

Leonhard Markus Helminger

Erika Varis Doggett

Alexander Sorkine Hornung

Jacek Krzysztof Naruniec

Jared McPhillen

Roberto Gerson De Albuquerque Azevedo

Henning Zimmer

Hayko Jochen Wilhelm Riemenschneider

Shinobu Hattori

Wenhao Zhang

Evan Goldberg

Mark Joseph Meyer

Olga Sorkine-Hornung

Yang Zhang

Joaquim Campos

Mauro Bamert

Derek Edward Bradley

Markus Gross

Federico Perazzi

Paulo Fabiano Urnau Gotardo

Brian McWilliams

Marios Papas

Michael J Bracco

Alexander Sorkine-Hornung

Stephan Marcel Mandt

Changil Kim

Mario Botsch

Simone Meyer

Yifan Wang

Karlis Martins Briedis

Ian Mcgonigal

Michael Bernasconi

Katja Wolff

Victor Cornillere

Nicholas David Gregory

Oliver Wang

Markus H Gross

Tunc Ozan Aydin

Jan Novak

Mark Raymond Mine

Thijs Vogels

Fabrice Pierre Armand Rousselle

Dane M Coffey

Daniel L Baker

Rasmus Tamstorf

David M Adler

Xuchang Huangfu

Charles Tappan

Jean-Charles Bazin

Daniel Teece

Lemei Huang

Siroberto Scerbo

James D Zimmerman

Stéphane Grabli

Marco Manzi

Mitchel C Jacobs

Farnood Salehi

Erika Elizabeth Varis Doggett

Henrik D Dahlberg

Gerhard Röthlin

Andre C Mazzone

Per H Christensen

Sirak Ghebremusse

Thomas Etterlin

Christopher Andreas Otto

Givi Meishvili

Jared P Mcphillen

Joan Massich

Antoine Milliez

Hayko Riemenschneider

Clara Fernandez

Zhenghao Chen

Lucas Relic

Cafer Mertcan Akcay

Eitan Abecassis

Joan Massich Vall

Caner Hazirbas

Katherine Hinsen

Clara Fernandez Labrador

Mingyang Song

Sutao Wang

Markus Plack

Raphael François Otriz

Mark A Meyer

Matthias Fauconneau

Simon Meierhans

Leonard Markus Helminger

Daniel Konrad Dorda

Chunk based video encoding involves dividing a video into chunks with varying lengths based on the content within those frames. In contrast, dividing the video at a fix interval is prone to generating chunks starting at the middle of hard to encode areas, which can lead to a loss of encoder rate-control efficiency and produce visual quality gaps at the beginning of such chunks. The system can identify a set of boundaries for dividing the video into chunks having similar lengths and with little to no impact on visual quality. The boundaries of the chunks are placed at locations (or frames) that are far from the complex (or hard to encode) areas of the video. To do so, the system evaluates the video using various complexity metrics to identify the complex areas that require more bits to encode relative to less complex areas.

Content adaptive boundary placement for distributed encodes

In some embodiments, a method sends information for a sample of content, a first question, and a second question for output on an interface. The first question receives, from a subject, a first response for a sample level rating for an artifact that is perceived to be visible in the sample and the second question receives, from the subject, a second response for regions in the sample that are perceived to contain the artifact. The method receives the first response for the sample level rating and the second response for regions that are perceived to contain the artifact. First responses are combined from multiple subjects to generate an opinion score for the sample and second responses are combined to generate region scores for regions. The method generates training data from the opinion score and the region scores to train a process to perform an action based on the artifacts.

Subjective quality assessment tool for image/video artifacts

A computer-implemented method of changing a face within an output image or video frame includes: receiving an input image that includes a face presenting a facial expression in a pose; separately encoding different portions of the image by, for each separately encoded portion, generating a latent space point of the portion, thereby generating a plurality of multi-dimensional vectors where each multi-dimensional vector is an encoded representation of a different portion of the input image; concatenating the plurality of multi-dimensional vectors into a combined latent space vector; and decoding the combined latent space vector to generate the output image in accordance with a desired facial identity but with the facial expression and pose of the face in the input image.

Method and system for deep learning based face swapping with multiple encoders

A system includes a hardware processor and a memory storing a video/audio (V/A) synchronizer including video and audio encoders. The hardware processor executes the V/A synchronizer to receive raw video and audio extracted from media content, partition the raw video into video frame patches, partition the raw audio into audio samples, pre-process the video frame patches and the audio samples for encoding. The hardware processor further executes the V/A synchronizer to encode, using the video encoder, the pre-processed video frame patches to provide pre-processed and encoded video frame patches used to provide a latent representation of the raw video, encode, using the audio encoder, the pre-processed audio samples to provide pre-processed and encoded audio samples used to provide a latent representation of the raw audio, and synchronize, using the latent representations of the raw video and the raw audio, the raw audio with the raw video.

Video and audio synchronization with dynamic frame and sample rates

Restoration methods and systems are disclosed for video remastering. Techniques disclosed include receiving a video sequence. For each frame of the video sequence, techniques disclosed include encoding, by a degradation encoder, a video content associated with the frame into a latent vector. The latent vector is a representation of the degradation present in the video content; the degradation present in the video content includes one or more degradation types. Based on the latent vector and the video content, techniques disclosed further include generating, by a backbone network, one or more feature maps, and, then, restoring the frame based on the one or more feature maps.

Deep learning framework for video remastering

Generating augmented reality content includes inputting a first layout of a physical space and a first set of anchor content into a machine learning model; generating, via execution of the machine learning model, a first augmented reality view that includes (i) a first portion of the physical space and (ii) an extension of the first set of anchor content across a second portion of the physical space; and causing the first augmented reality view to be outputted in a computing device.

Neural extension of 3D content in augmented reality environments

One embodiment of the present invention sets forth a technique for inserting content into a media program. The technique includes determining a plurality of markers corresponding to a plurality of locations within a media program. The technique also includes for each marker included in the plurality of markers, automatically analyzing a first set of intervals within the media program that lead up to the marker and a second set of intervals within the media program that immediately follow the marker and determine a set of audio characteristics associated with the first set of intervals and the second set of intervals. The technique further includes generating a plurality of scores for the plurality of markers based on the set of audio characteristics for each marker and inserting additional content at one or more markers included in the plurality of markers based on the plurality of scores.

Optimizing insertion points for content based on audio and video characteristics

A system includes a machine learning (ML) model-based video encoder configured to receive an uncompressed video sequence including multiple video frames, determine, from among the multiple video frames, a first video frame subset and a second video frame subset, encode the first video frame subset to produce a first compressed video frame subset, and identify a first decompression data for the first compressed video frame subset. The ML model-based video encoder is further configured to encode the second video frame subset to produce a second compressed video frame subset, and identify a second decompression data for the second compressed video frame subset. The first decompression data is specific to decoding the first compressed video frame subset but not the second compressed video frame subset, and the second decompression data is specific to decoding the second compressed video frame subset but not the first compressed video frame subset.

Microdosing for low bitrate video compression

Certain aspects of the present disclosure provide techniques for adaptive sampling for rendering using deep learning. This includes receiving, at a sampler in a rendering pipeline, a plurality of rendered pixel data, wherein the sampler includes a first machine learning (ML) model. It further includes generating a sampling map for the rendering pipeline using the first ML model and the plurality of rendered pixel data, including predicting a plurality of pixel values in the sampling map based on a generated distribution of pixel values. It further includes rendering an image using the sampler, the sampling map, and a denoiser in the rendering pipeline.

Adaptive sampling using deep learning

One embodiment of the present invention sets forth a technique for performing face swapping. The technique includes converting a first input image that depicts a first facial identity from a first viewpoint at a first time into a first latent representation and converting a second input image that depicts the first facial identity from a second viewpoint at the first time into a second latent representation. The technique also includes generating, via a first machine learning model, a first output image that depicts a second facial identity from the first viewpoint based on the first latent representation. The technique further includes generating, via the first machine learning model, a second output image that depicts the second facial identity from the second viewpoint based on the second latent representation.

Multi-camera face swapping

A system includes a processor and a memory storing software code. The processor executes the software code to receive a first rendered image having first color and auxiliary buffers, receive a second rendered image having second color and auxiliary buffers, provide a third auxiliary buffer for an intermediate image to be interpolated, and identify one or more context features from each of the color buffers and the auxiliary buffers. The software code further interpolates, based on the context features, the intermediate image to provide an interpolated image, determines, based on the one or more context features from each of the first, second, and third auxiliary buffers, a rendering mask for producing one or more rendered sub-portions of the intermediate image, partially renders, using the rendering mask, the intermediate image to provide a partially rendered image, and blends the interpolated image and the partially rendered image to provide a composite image.

Kernel-based frame interpolation for spatio-temporally adaptive rendering

In some embodiments, a system includes a first component to extract temporal features from a current frame being coded and a previous frame of a video. A second component uses a first transformer to fuse spatial features from the current frame with the temporal features to generate spatio-temporal features as first output. A third component uses a second transformer to perform entropy coding using the first output and at least a portion of the temporal features to generate a second output. A fourth component uses a third transformer to reconstruct the current frame based on the first output that is processed using the second output and the temporal features.

Contextual video compression framework with spatial-temporal cross-covariance transformers

One embodiment of the present invention sets forth a technique for performing frame interpolation. The technique includes generating (i) a first set of feature maps based on a first set of rendering features associated with a first key frame, (ii) a second set of feature maps based on a second set of rendering features associated with a second key frame, and (iii) a third set of feature maps based on a third set of rendering features associated with a target frame. The technique also includes applying one or more neural networks to the first, second, and third set of feature maps to generate a set of mappings from a first set of pixels in the first key frame to a second set of pixels in the target frame. The technique further includes generating the target frame based on the set of mappings.

Frame interpolation for rendered content

In some embodiments, a method trains a first parameter of a differentiable proxy codec to encode source content based on a first loss between first compressed source content and second compressed source content that is output by a target codec. A pre-processor pre-processes a source image to output a pre-processed source image, the pre-processing being based on a second parameter. The differentiable proxy codec encodes the pre-processed source image into a compressed pre-processed source image based on the first parameter. The method determines a second loss between the source image and the compressed pre-processed source image and determines an adjustment to the first parameter based on the second loss. The adjustment is used to adjust the second parameter of the pre-processor based on the second loss.

Performance optimization of pre-processor using video proxy codec

A system includes a machine learning (ML) model-based video downsampler configured to receive an input video sequence having a first display resolution, and to map the input video sequence to a lower resolution video sequence having a second display resolution lower than the first display resolution. The system also includes a neural network-based (NN-based) proxy video codec configured to transform the lower resolution video sequence into a decoded proxy bitstream. In addition, the system includes an upsampler configured to produce an output video sequence using the decoded proxy bitstream.

Codec rate distortion compensating downsampler

Growing community of inventors

Zurich, Switzerland

Christopher Richard Schroers