Ssimwave Inc.

Imax Corporation

Other

Zhou Wang

Kai Zeng

Abdul Rehman

Ahmed Badr

Hojatollah Yeganeh

Jiheng Wang

Peter Olijnyk

Ian Wormsbecker

Christopher Vytautas Olekas

Christopher Kulbakas

Kaiwen Ye

Waqas Ahmed

Zhengfang Duanmu

A method or system is disclosed to assess transmission errors in a visual media input. Domain knowledge is obtained from the visual media input by content analysis, codec analysis, distortion analysis, and human visual system modeling. The visual media input is divided into partitions, which are passed into deep neural networks (DNNs). The DNN outputs of all partitions are combined with the guidance of domain knowledge to produce an assessment of the transmission error. In one or more illustrative examples, transmission error assessment at a plurality of monitoring points in a visual media communication system is collected and assessed, followed by quality control processes and statistical performance assessment on the stability of the visual communication system.

Automatic visual media transmission error assessment

No-reference (NR) quality assessment (VQA) of a test visual media input encoding media content is provided. The test video visual media input is decomposed into multiple-channel representations. Domain knowledge is obtained by performing content analysis, distortion analysis, human visual system (HVS) modeling, and/or viewing device analysis. The multiple-channel representations are passed into deep neural networks (DNNs) producing DNN outputs. The DNN outputs are combined using domain knowledge to produce an overall quality score of the test visual media input.

No-reference visual media assessment combining deep neural networks and models of human visual system and video content/distortion analysis

Correlated quality-of-experience (QoE) and latency measures are generated at a plurality of monitoring points along a multimedia delivery chain including multiple video operations. At each of the plurality of monitoring points, an absolute QoE measure defined on a human perceptual quality scale for media content is computed, and one or more of content extraction or feature extraction on the media content are performed. To a common middleware from each of the plurality of monitoring points, the respective QoE measure and results of the one or more of content extraction or feature extraction are transmitted. Computing and updating an absolute QoE measure for each of the plurality of monitoring points is performed. Computing and updating latencies between multiple monitoring points at the middleware using the results from each of the plurality of monitoring points is also performed.

Unified end-to-end quality and latency measurement, optimization and management in multimedia communications

Candidate encodes of a source video are produced using a plurality of encoding configurations. Quality of experience (QoE) scores for each of the candidate encodes are computed. The QoE scores of the plurality of candidate encodes are compared to determine a lowest bitrate encode that achieve a target QoE score. The lowest bitrate encode that achieve the target QoE score as an optimized output video is selected.

Perceptual quality assessment metric based content and viewer aware encoding

Macroblocking artifacts are assessed in an encoded video. Edges are detected and edge strengths are computed at macroblock boundaries in each decoded frame of the video. The detected edges are paired at horizontal or vertical sides of each macroblock to identify macroblock edge pairs. A macroblocking level of the frame is identified according to the macroblock edge pairs. For instance, for each of the macroblock edge pairs, the edges of the pair are identified as being macroblocking edges if edge strengths of both sides of the pair exceed a significance threshold, and the macroblocking level of the frame is determined by aggregating the macroblocking edges in the frame.

Macroblocking artifact detection

Frames of a video stream collected at first and second points along a video delivery chain are buffered into first and second buffers, respectively, the second point being downstream the video delivery chain from the first point. A control group is identified as a subset of frames of the second buffer. Correlations of extracted features of the control group to extracted features of successive windows of frames of the first buffer are computed, the extracted features being based on spatial information and temporal information of the video stream. A delay between the video stream collected at the first point and the video stream collected at the second point is identified according to a maximum correlation of the correlations.

Real-time latency measurement of video streams

Banding effects in an image or video are assessed. The image or each frame of a video is decomposed into luminance or color channels. Signal activity is computed at each spatial location in each channel. A significance of the signal activity at each spatial location is determined by comparing each element of the signal activity with a significance threshold. Banding pixels are detected as those pixels of the image or frame having significant signal activity at their respective spatial locations and having non-significant signal activity of at least a minimum threshold percentage of neighboring pixels to the respective spatial locations.

Image and video banding assessment

The present invention is a system or method that facilitates smart decision makings at the client side for adaptive video streaming over a video delivery network by making use of perceptual video quality-of-experience predictions performed during the video preparation stage, at the video hosting or cache server side, or inside the video delivery network, and then transmitted to the client. Compared with prior art approaches of adaptive bitrate video streaming, the present invention can result in one or more of the following benefits: 1) save the overall bandwidth for the delivery of the video content without scarifying the client users' quality-of-experience; 2) create better overall visual quality-of-experience of the client users; 3) create smoother visual quality-of-experience of the client users; and 4) reduce the probability of rebuffering or stalling events at the client user side.

Method and system for smart adaptive video streaming driven by perceptual quality-of-experience estimations

Disclosed are system and method that automate measurement of end users' quality-of-experience (QoE) when perceiving the video being streamed to the users' viewing devices. The overall user QoE is measured and computed by combining the instantaneous presentation quality, the playback smoothness quality, and the interactions between them. Prediction accuracy is thus significantly improved. The instantaneous and end-of-process QoE measures created by the system and method described are suitable for the monitoring and optimization of media streaming systems and services.

Method and system for automatic user quality-of-experience measurement of streaming video

There is disclosed a method and system for objective video quality assessment. The objective video quality assessment approach automatically predicts human visual quality assessment behaviors in evaluating videos that contain artifacts generated during video acquisition, storage, reproduction, compression, transmission, processing, and/or display. The present method and system computes a five-dimensional quality map of a video being assessed, where the map indicates the local quality variations of the video in a five-dimensional space, which includes two spatial dimensions, one scale dimension, one time dimension, and one distortion type dimension. The present method and system may use one or a combination of device and viewing condition parameters in the generation of the quality map, and in the combination of the quality map into a scalar or vector-values measure that indicates quality aspects of the test videos.

Method and system for objective perceptual video quality assessment

The present invention is a system and method for video coding. The video coding system may involve a structural similarity-based divisive normalization approach, wherein the frame prediction residual of the current frame may be transformed to form a set of coefficients and a divisive normalization mechanism may be utilized to normalize each coefficient. The normalization factor may be designed to reflect or approximate the normalization factor in a structural similarity definition. The Lagrange parameter for RDO for divisive normalization coefficients may be determined by both the quantization step and a prior distribution function of the coefficients. The present invention may generally be utilized to improve the perceptual quality of decoded video without increasing data rate, or to reduce the data rate of compressed video stream without sacrificing the perceived quality of decoded video. The present invention has shown to significantly improve the coding efficiency of MPEG4/H.264 AVC and HEVC coding schemes. The present invention may be utilized to create video codes compatible with prior art and state-of-the-art video coding standards such as MPEG4/H.264 AVC and HEVC. The present invention may also be utilized to create video codecs incompatible with existing standards, so as to further improve the coding gain.

Method and system for structural similarity based perceptual video coding

There is disclosed a system and method for video coding, and more particularly to video coding that uses structural similarity (SSIM) based rate-distortion optimization methods to improve the perceptual quality of decoded video without increasing data rate, or to reduce the data rate of compressed video stream without sacrificing perceived quality of the decoded video. In an embodiment, the video coding system and method may be a SSIM-based rate-distortion optimization approach that involves minimizing a joint cost function defined as the sum of a data rate term and a distortion functions. The distortion function may be defined to be monotonically increasing with the decrease of SSIM and a Lagrange parameter may be utilized to control the trade-off between rate and distortion. The optimal Lagrange parameter may be found by utilizing the ratio between a reduced-reference SSIM model with respect to quantization step, and a data rate model with respect to quantization step. In an embodiment, a group-of-picture (GOP) level quantization parameter (QP) adjustment method may be used in multi-pass encoding to reduce the bit-rate while keeping similar perceptual video quality. In another embodiment, a frame level QP adjustment method may be used in single-pass encoding to achieve constant SSIM quality. In accordance with an embodiment, the present invention may be implemented entirely at the encoder side and may or may not require any change at the decoder, and may be made compatible with existing video coding standards.

Growing community of inventors

Waterloo, Canada

Zhou Wang