Euclid Discoveries, LLC

Avid Technology Limited

Dane P Kottke

Nigel Lee

Jeyun Lee

Katherine H Cornog

Myo Tun

John J Guo

Sangseok Park

Michel Rynderman

Christopher Weed

Justin Kwan

Videos may be characterized by objective metrics that quantify video quality. Embodiments are directed to target bitrate prediction methods in which one or more objective metrics may serve as inputs into a model that predicts a mean opinion score (MOS), a measure of perceptual quality, as a function of metric values. The model may be derived by generating training data through conducting subjective tests on a set of video encodings, obtaining MOS data from the subjective tests, and correlating the MOS data with metric measurements on the training data. The MOS predictions may be extended to predict the target (encoding) bitrate that achieves a desired MOS value. The target bitrate prediction methods may be applied to segments of a video. The methods may be made computationally faster by applying temporal subsampling. The methods may also be extended for adaptive bitrate (ABR) applications by applying scaling factors to predicted bitrates at one frame size to determine predicted bitrates at different frame sizes. A dynamic scaling algorithm may be used to determine predicted bitrates at different frame sizes.

Selection of video quality metrics and models to optimize bitrate savings in video encoding applications

Videos may be characterized by objective metrics that quantify video quality. Embodiments are directed to target bitrate prediction methods in which one or more objective metrics may serve as inputs into a model that predicts a mean opinion score (MOS), a measure of perceptual quality, as a function of metric values. The model may be derived by generating training data through conducting subjective tests on a set of video encodings, obtaining MOS data from the subjective tests, and correlating the MOS data with metric measurements on the training data. The MOS predictions may be extended to predict the target (encoding) bitrate that achieves a desired MOS value. The target bitrate prediction methods may be applied to segments of a video. The methods may be made computationally faster by applying temporal subsampling. The methods may also be extended for adaptive bitrate (ABR) applications by applying scaling factors to predicted bitrates at one frame size to determine predicted bitrates at different frame sizes. A dynamic scaling algorithm may be used to determine predicted bitrates at the different frame sizes.

Dynamic scaling for consistent video quality in multi-frame size encoding

Video characterization for smart encoding based on perceptual quality optimization

Perceptual statistics are used to compute importance maps that indicate which regions of a video frame are important to the human visual system. Importance maps may be generated from encoders that produce motion vectors and employ motion estimation for inter-prediction. The temporal contrast sensitivity function (TCSF) may be computed from the encoder's motion vectors. Quality metrics may be used to construct a true motion vector map (TMVM), which refines the TCSF. Spatial complexity maps (SCMs) can be calculated from simple metrics (e.g. block variance, block luminance, SSIM, and edge detection). Importance maps with TCSF, TMVM, and SCM may be used to modify the standard rate-distortion optimization criterion for selecting the optimum encoding solution. Importance maps may modify encoder quantization. The spatial information for the importance maps may be provided by a lookup table based on block variance, where negative and positive spatial QP offsets for block variances are provided.

Perceptual optimization for model-based video encoding

Perceptual statistics may be used to compute importance maps that indicate which regions of a video frame are important to the human visual system. Importance maps may be applied to the video encoding process to enhance the quality of encoded bitstreams. The temporal contrast sensitivity function (TCSF) may be computed from the encoder's motion vectors. Motion vector quality metrics may be used to construct a true motion vector map (TMVM) that can be used to refine the TCSF. Spatial complexity maps (SCMs) can be calculated from metrics such as block variance, block luminance, SSIM, and edge strength, and the SCMs can be combined with the TCSF to obtain a unified importance map. Importance maps may be used to improve encoding by modifying the criterion for selecting optimum encoding solutions or by modifying the quantization for each target block to be encoded.

Continuous block tracking (CBT) tracks macroblock locations over reference frames to produce better inter-predictions than conventional block-based motion estimation/compression. CBT includes frame-to-frame tracking, estimating motion from a frame to a previous frame, and continuous tracking, related frame-to-frame motion vectors to block tracks. Frame-to-frame tracking may include block based or hierarchical motion estimations. CBT combined with enhanced predictive zonal search may create unified motion estimation. Accumulated CBT results may form trajectories for trajectory-based CBT predictions. Metrics measuring continuous track and motion vector quality can assess relative priority of CBT prediction against non-tracker-based predictions and to modify encoding choices. Continuous tracks can be analyzed for goodness-of-fit to translational motion models, with outliers removed from encoding. Translational motion models can be extended to entire frames in adaptive picture type selection. Outputs from CBT used in look-ahead processing, via look-ahead tracking, may provide rate control and scene change detection for the current frame being encoded.

Continuous block tracking for temporal prediction in video encoding

High quality intraframe-only compression of video can be achieved using rate distortion optimization and without resizing or bit depth modification. The compression process involves transforming portions of the image to generate frequency domain coefficients for each portion. A bit rate for each transformed portion using a plurality of scale factors is determined. Distortion for each portion is estimated according to the plurality of scale factors. A scale factor is selected for each portion to minimize the total distortion in the image to achieve a desired bit rate. A quantization matrix is selected according to the desired bit rate. The frequency domain coefficients for each portion are quantized using the selected plurality of quantizers as scaled by the selected scale factor for the portion. The quantized frequency domain coefficients are encoded using a variable length encoding to provide compressed data for each of the defined portions. The compressed data is output for each of the defined portions to provide a compressed bitstream at the desired bit rate.

Growing community of inventors

Durham, NC, United States of America

Dane P Kottke