Tencent America LLC

Ding Ding

Shan Liu

Wei Jiang

Xiaozhong Xu

Wei Wang

Wei W Wang

Sheng Lin

Xin Zhao

Roman Chernyak

This disclosure relates generally to video coding/decoding and particularly for evaluating codec performance. One method includes obtaining a first plurality of anchor data points; obtaining a second plurality of test data points; analyzing monotonicity conditions of the first plurality of anchor data points and the second plurality of test data points; determining a fitting algorithm based on the analyzed monotonicity conditions; fitting the first plurality of anchor data points with the fitting algorithm to obtain an anchor curve; fitting the second plurality of test data points with the fitting algorithm to obtain a test curve; and evaluating the test codec performance based on the anchor curve and the test curve, to obtain an evaluation result.

Methods and devices for video codec performance measurement and evaluation

Aspects of the disclosure provide a method, an apparatus, and non-transitory computer-readable storage medium for video decoding. The apparatus includes processing circuitry that reconstructs blocks of an image that is to be reconstructed from a coded video bitstream. The processing circuitry decodes first deblocking information in the coded video bitstream including a first deblocking parameter of a deep neural network (DNN) in a video decoder. The first deblocking parameter of the DNN is an updated parameter that has been previously determined by a content adaptive training process. The processing circuitry determines the DNN for a first boundary region comprising a subset of samples in the reconstructed blocks based on the first deblocking parameter included in the first deblocking information. The processing circuitry deblocks the first boundary region comprising the subset of samples in the reconstructed blocks based on the determined DNN corresponding to the first deblocking parameter.

Content-adaptive online training method and apparatus for deblocking in block-wise image compression

Aspects of the disclosure provide methods and apparatuses, for video decoding and encoding. The apparatus includes processing circuitry configured to receive an image/video comprising one or more blocks and metadata for a machine task associated with the image/video. The metadata specifies neural network post-filtering characteristics for machine consumption. The processing circuitry decodes a first post-filtering parameter in the image/video corresponding to the one or more blocks to be reconstructed. The first post-filtering parameter applies to a block in the one or more blocks and has been updated by a post-filtering module in a post-filtering neural network (NN) that is trained based on a training dataset and the metadata. The processing circuitry determines the post-filtering NN in a video decoder corresponding to the one or more blocks based on the first post-filtering parameter, and decodes the block based on the determined post-filtering NN corresponding to the block and the metadata.

Content-adaptive online training method and apparatus for post-filtering

Methods, apparatus, and computer readable storage medium evaluating codec performance. One method includes obtaining m anchor data points each generated based on a respective anchor encoded video bitstream; obtaining n test data points each generated based on a respective encoded test video bitstream, n being an integer; fitting the m anchor data points with an anchor curve, the anchor curve being based on an anchor polynomial, wherein the anchor polynomial is monotonic in an x-axis range; fitting the n test data points with a test curve, the anchor curve being based on a test polynomial, wherein the test polynomial is monotonic in the x-axis range; and evaluating the test codec performance based on the anchor curve and the test curve, to obtain an evaluation result.

Method and apparatus for video coding performance assessment

Neural network based image compression using an optimized end-to-end (E2E) neural image compression (NIC) framework being performed by at least one processor and includes receiving an input image to an E2E NIC framework, processing the input image in its entirety using an encoder in the E2E NIC framework to obtain a first bitstream representation of the input image in its entirety, reconstructing an output image from the first bitstream representation of the input image in its entirety using a decoder in the E2E NIC framework, optimizing the encoder in the E2E NIC framework by reducing a distortion loss between the input image and the output image, and processing the input image using the optimized encoder in the E2E NIC framework to obtain a second bitstream representation of the input image. Further, the input may be partitioned into blocks for which encoder optimization is performed for each block and each block is encoded instead of the entire image.

Method and apparatus for optimizing end-to-end neural image compression framework

Content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, is provided, including receiving an input image, to an E2E NIC framework, including one or more blocks, preprocessing a first neural network of the E2E NIC framework, based on the one or more blocks, computing updated parameters using the preprocessed first neural network, encoding the one or more blocks and the updated parameters, updating the first neural network based on the encoded updated parameters, and generating a compressed representation of the encoded one or more blocks using the updated first neural network.

System, method, and computer program for content adaptive online training for multiple blocks in neural image compression

In some examples, processing circuitry decodes, from a coded bitstream that carries a compressed image, an index that points a value in a set of values of a parameter. A value change of the parameter adjusts a compression rate of the compressed image. The compressed image is generated by a neural network based encoder based on the parameter. The processing circuitry inputs the value of the parameter into a multi-rate compression domain computer vision task decoder. The multi-rate compression domain computer vision task decoder includes one or more neural networks for performing a computer vision task from compressed images according to corresponding values of the parameter that are used for generating the compressed images. The multi-rate compression domain computer vision task decoder generates a computer vision task result according to the compressed image in the coded bitstream and the value of the parameter.

Multi-rate of computer vision task neural networks in compression domain

Neural network based substitutional end-to-end (E2E) image compression (NIC) being performed by at least one processor and includes receiving an input image to an E2E NIC framework, determining a step size of the input image indicating a learning rate of a training model, determining a substitute image based on the training model, encoding the substitute image in lieu of the input image to generate a bitstream, and mapping the substitute image to the bitstream to generate a compressed representation. Further, step size may be determined by a scheduler and change throughout the training of the training model. The image may also be split into patches for which a scheduler is assigned for each patch and each patch is encoded instead of the entire input image.

Method and apparatus for dynamic learning rates of substitution in neural image compression

Neural network based substitutional end-to-end (E2E) image compression (NIC) being performed by at least one processor and includes receiving an input image to an E2E NIC framework, determining a substitute image based on a training model of the E2E NIC framework, encoding the substitute image to generate a bitstream, mapping the substitute image to the bitstream to generate a compressed representation of the input image. Further, the input may be partitioned into blocks for which a substitute representation is determined for each block and each block is encoded instead of the entire substitute image.

Method and apparatus for multi-learning rates of substitution in neural image compression

A method and apparatus for reducing artifacts in a compressed image using a neural-network based deblocking filter. The method may include receiving at least one reconstructed image, wherein each reconstructed image comprises one or more reconstructed blocks and extracting boundary areas associated with boundaries of the one or more reconstructed blocks in the at least one reconstructed image. The extracted boundary areas may be input in a trained deblocking model to generate boundary areas having reduced artifacts and the trained deblocking mode is trained on training data based on estimated compression by a neural image compression (NIC) network. The edge areas associated with the generated boundary areas may be removed; and at least one reconstructed image with reduced artifacts may be generated based on the generated boundary areas.

Neural network-based deblocking filters

A method of block-wise neural image compression with post filtering is performed by at least one processor of an encoder and includes encoding a block of an input image, using a first neural network, wherein the encoded block is decoded by a decoder using a second neural network to generate a reconstructed block, and performing intra-prediction on the reconstructed block, using a third neural network, to generate a predicted block. The method further includes determining a difference between the block of the input image and the generated predicted block, to generate a prediction residual, encoding the generated prediction residual, using a fourth neural network, wherein the encoded prediction residual is decoded by the decoder using a fifth neural network, and adding the decoded prediction residual to the generated predicted block, to generate a recovered predicted block.

Method and apparatus for block-wise neural image compression with post filtering

Aspects of the disclosure provide a method, an apparatus, and a non-transitory computer-readable storage medium for video decoding. The apparatus can include processing circuitry. The processing circuitry is configured to reconstruct blocks of an image from a coded video bitstream. The processing circuitry can perform post-processing on one of a plurality of regions of first two neighboring reconstructed blocks of the reconstructed blocks with at least one post-processing neural network (NN). The first two neighboring reconstructed blocks have a first shared boundary and include a boundary region having samples on both sides of the first shared boundary. The plurality of regions of the first two neighboring reconstructed blocks includes the boundary region and non-boundary regions that are outside the boundary region. The one of the plurality of regions is replaced with the post-processed one of the plurality of regions of the first two neighboring reconstructed blocks.

Block-wise content-adaptive online training in neural image compression with post filtering

Aspects of the disclosure provide a method, an apparatus, and a non-transitory computer-readable storage medium for video decoding. The apparatus can include processing circuitry. The processing circuitry is configured to receive an image or video comprising one or more blocks. The processing circuitry can decode a first post-filtering parameter in the image or video corresponding to the one or more blocks to be reconstructed. The first post-filtering parameter applies to at least one of the one or more blocks and has been updated by a post-filtering module in a post-filtering neural network (NN) that is trained based on a training dataset. The processing circuitry can determine the post-filtering NN in a video decoder corresponding to the one or more blocks based on the first post-filtering parameter. The processing circuitry can decode the one or more blocks based on the determined post-filtering NN corresponding to the one or more blocks.

A method and apparatus for neural network based cross component prediction with scaling factors during encoding or decoding of an image frame or a video sequence, which may include training a deep neural network (DNN) cross component prediction (CCP) model with at least one or more scaling factors, wherein the at least one or more scaling factors are learned by optimizing a rate-distortion loss based on an input video sequence comprising a luma component, and reconstructing a chroma component based on the luma component using the trained DNN CCP model with the at least one or more scaling factors for chroma prediction. The trained DNN CCP may be updated for chroma prediction of the input video sequence using the one or more scaling factors, and performing chroma prediction of the input video sequence using the updated DNN CCP model with the one or more scaling factors.

Content-adaptive online training for DNN-based cross component prediction with scaling factors

Aspects of the disclosure provide a method and an apparatus for video encoding. The apparatus includes processing circuitry configured to generate an initial feature representation from an input image to be encoded and perform an iterative update of values of a plurality of elements in the initial feature representation. The iterative update includes generate a coded representation corresponding to a final feature representation based on the final feature representation that has been updated from the initial feature representation by a number of iterations of the iterative update. A reconstructed image corresponding to the final feature representation is generated based on the coded representation. An encoded image corresponding to the final feature representation having updated values of the plurality of elements is generated. One of (i) a rate-distortion loss corresponding to the final feature representation or (ii) the number of iterations of the iterative update satisfies a pre-determined condition.

Growing community of inventors

Palo Alto, CA, United States of America

Ding Ding