Netflix Inc.

The University of Texas System

Intel Corporation

Christos G Bampis

Zhi Li

Anush Moorthy

Anne Aaron

Li-Heng Chen

Alan C Bovik

Jin Soo Park

Lavanya Sharan

Julie Novak

Sungsoo Kim

Georgios Alex Dimakis

Martin Tingley

Gowri Somanath

Aditya Mavlankar

In various embodiments an endpoint application reconstructs downscaled videos. The endpoint application accesses metadata associated with a portion of a downscaled video that has a first resolution and was generated using a trained downscaling convolutional neural network (CNN). The endpoint application determines, based on the metadata, an upscaler that should be used when upscaling the portion of the downscaled video. The endpoint application executes the upscaler on the portion of the downscaled video to generate a portion of a reconstructed video that is accessible for playback and has a second resolution that is greater than the first resolution.

Techniques for reconstructing downscaled video content

In various embodiments, a training application generates a trained perceptual quality model that estimates perceived video quality for reconstructed video. The training application computes a pixels-per-degree value based on a normalized viewing distance and a display resolution. The training application computes a set of feature values corresponding to a set of visual quality metrics based on a reconstructed video sequence, a source video sequence, and the pixels-per-degree value. The training application executes a machine learning algorithm on the first set of feature values to generate the trained perceptual quality model. The trained perceptual quality model maps a particular set of feature values corresponding to the set of visual quality metrics to a particular perceptual quality score.

Techniques for generating a perceptual quality model for predicting video quality across different viewing parameters

In various embodiments, a quality inference application estimates perceived video quality for reconstructed video. The quality inference application computes a set of feature values corresponding to a set of visual quality metrics based on a reconstructed frame, a source frame, a display resolution, and a normalized viewing distance. The quality inference application executes a trained perceptual quality model on the set of feature values to generate a perceptual quality score that indicates a perceived visual quality level for the reconstructed frame. The quality inference application performs one or more operations associated with an encoding process based on the perceptual quality score.

Techniques for predicting video quality across different viewing parameters

In various embodiments, a bootstrapping training subsystem performs sampling operation(s) on a training database that includes subjective scores to generate resampled dataset. For each resampled dataset, the bootstrapping training subsystem performs machine learning operation(s) to generate a different bootstrap perceptual quality model. The bootstrapping training subsystem then uses the bootstrap perceptual quality models to quantify the accuracy of a perceptual quality score generated by a baseline perceptual quality model for a portion of encoded video content. Advantageously, relative to prior art solutions in which the accuracy of a perceptual quality score is unknown, the bootstrap perceptual quality models enable developers and software applications to draw more valid conclusions and/or more reliably optimize encoding operations based on the perceptual quality score.

Quantifying perceptual quality model uncertainty via bootstrapping

A computer-implemented method, system and computer program product for compressing video. A set of video frames is partitioned into two subsets of different types of frames, a first type and a second type. The first type of frames of videos is compressed to generate a first representation by a first stage encoder. The first representation is then decoded to reconstruct the first type of frames using a first stage decoder. The second type of frames of video is compressed to generate a second representation that only contains soft edge information by a second stage encoder. A generative model corresponding to a second stage decoder is then trained using the first representation and the reconstructed first type of frames by using a discriminator employed by a machine learning system. After training the generative model, it generates reconstructed first and second types of frames using the soft edge information.

Video compression technique using a machine learning system

In various embodiments, a training application trains a convolutional neural network to downsample images in a video encoding pipeline. The convolution neural network includes at least two residual blocks and is associated with a downsampling factor. The training application executes the convolutional neural network on a source image to generate a downsampled image. The training application then executes an upsampling algorithm on the downsampled image to generate a reconstructed image having the same resolution as the source image. The training application computes a reconstruction error based on the reconstructed image and the source image. The training application updates at least one parameter of the convolutional neural network based on the reconstruction error to generate a trained convolutional neural network. Advantageously, the trained convolution neural network can be implemented in a video encoding pipeline to mitigate visual quality reductions typically experienced with conventional video encoding pipelines that implement conventional downsampling techniques.

Machine learning techniques for video downsampling

In various embodiments, a perceptual quality application determines an absolute quality score for encoded video content viewed on a target viewing device. In operation, the perceptual quality application determines a baseline absolute quality score for the encoded video content viewed on a baseline viewing device. Subsequently, the perceptual quality application determines that a target value for a type of the target viewing device does not match a base value for the type of the baseline viewing device. The perceptual quality application computes an absolute quality score for the encoded video content viewed on the target viewing device based on the baseline absolute quality score and the target value. Because the absolute quality score is independent of the viewing device, the absolute quality score accurately reflects the perceived quality of a wide range of encoded video content when decoded and viewed on a viewing device.

Device-consistent techniques for predicting absolute perceptual video quality

In various embodiments, a prediction application computes a quality score for re-constructed visual content that is derived from visual content. The prediction application generates a frame difference matrix based on two frames included in the re-constructed video content. The prediction application then generates a first entropy matrix based on the frame difference matrix and a first scale. Subsequently, the prediction application computes a first value for a first temporal feature based on the first entropy matrix and a second entropy matrix associated with both the visual content and the first scale. The prediction application computes a quality score for the re-constructed video content based on the first value, a second value for a second temporal feature associated with a second scale, and a machine learning model that is trained using subjective quality scores. The quality score indicates a level of visual quality associated with streamed video content.

Techniques for modeling temporal distortions when predicting perceptual video quality

In various embodiments, a quality of experience (QoE) prediction application computes a visual quality score associated with a stream of encoded video content. The QoE prediction application also determines a rebuffering duration associated with the stream of encoded video content. Subsequently, the QoE prediction application computes an overall QoE score associated with the stream of encoded video content based on the visual quality score, the rebuffering duration, and an exponential QoE model. The exponential QoE model is generated using a plurality of subjective QoE scores and a linear regression model. The overall QoE score indicates a quality level of a user experience when viewing reconstructed video content derived from the stream of encoded video content.

Machine learning techniques for determining quality of user experience

In various embodiments, a training application trains a machine learning model to preprocess images. In operation, the training application computes a chroma sampling factor based on a downscaling factor and a chroma subsampling ratio. The training application executes a machine learning model that is associated with the chroma sampling factor on data that corresponds to both an image and a first chroma component to generate preprocessed data corresponding to the first chroma component. Based on the preprocessed data, the training application updates at least one parameter of the machine learning model to generate a trained machine learning model that is associated with the first chroma component.

Machine learning techniques for component-based image preprocessing

In various embodiments, a training application generates a perceptual video model. The training application computes a first feature value for a first feature included in a feature vector based on a first color component associated with a first reconstructed training video. The training application also computes a second feature value for a second feature included in the feature vector based on a first brightness component associated with the first reconstructed training video. Subsequently, the training application performs one or more machine learning operations based on the first feature value, the second feature value, and a first subjective quality score for the first reconstructed training video to generate a trained perceptual quality model. The trained perceptual quality model maps a feature value vector for the feature vector to a perceptual quality score.

Techniques for training a perceptual quality model to account for brightness and color distortions in reconstructed videos

In various embodiments, a quality inference application estimates the perceived quality of reconstructed videos. The quality inference application computes a first feature value for a first feature included in a feature vector based on a color component associated with a reconstructed video. The quality inference application also computes a second feature value for a second feature included in the feature vector based on a brightness component associated with the reconstructed video. Subsequently, the quality inference application computes a perceptual quality score based on the first feature value and the second feature value. The perceptual quality score indicates a level of visual quality associated with at least one frame included in the reconstructed video.

Techniques for computing perceptual video quality based on brightness and color components

In various embodiments, a perceptual quality application computes an absolute quality score for encoded video content. In operation, the perceptual quality application selects a model based on the spatial resolution of the video content from which the encoded video content is derived. The model associates a set of objective values for a set of objective quality metrics with an absolute quality score. The perceptual quality application determines a set of target objective values for the objective quality metrics based on the encoded video content. Subsequently, the perceptual quality application computes the absolute quality score for the encoded video content based on the selected model and the set of target objective values. Because the absolute quality score is independent of the quality of the video content, the absolute quality score accurately reflects the perceived quality of a wide range of encoded video content when decoded and viewed.

Growing community of inventors

Austin, TX, United States of America

Christos G Bampis