Tencent America LLC

Renesas Electronics America Inc.

Intel Corporation

Wei Wang

Shan Liu

Wei Jiang

Xiaozhong Xu

Sheng Lin

Ding Ding

Byeongdoo Choi

James Wang

Zeqiang Li

Jankin Hu

Liqiang Wang

Shubing Zhai

Elmoustapha Ould-Ahmed-Vall

Stephan Wenger

Eriko Nurvitadhi

Amit Bleiweiss

Deborah T Marr

Jeffrey J Cook

Srinivas Sridharan

Asit K Mishra

Dheevatsa Mudigere

Dhawal Srivastava

Mohammad Ashraf Bhuiyan

Anavai Ramesh

Niharika Maheshwari

Md Faijul Amin

A method and apparatus for adaptive decoding of compressed video for video conferencing may be provided. The method may include receiving compressed video data comprising a plurality of video frames, and determining, a selection signal indicating whether at least one of a face restoration technique and a face reenactment technique is to be used. The method may include adaptively selecting and transmitting a single reference frame or a plurality of low resolution (LR) frames comprising essential facial features, generating, one or more recovered facial features and one or more respective decompressed low resolution (LR) extended face areas based on the selection signal and the compressed video data, and decoding a video frame from the plurality of video frames based on the one or more recovered facial features and the one or more respective decompressed low resolution (LR) extended face areas.

Video conferencing based on adaptive face re-enactment and face restoration

Aspects of the disclosure provide a method, an apparatus, and non-transitory computer-readable storage medium for video decoding. The apparatus includes processing circuitry that reconstructs blocks of an image that is to be reconstructed from a coded video bitstream. The processing circuitry decodes first deblocking information in the coded video bitstream including a first deblocking parameter of a deep neural network (DNN) in a video decoder. The first deblocking parameter of the DNN is an updated parameter that has been previously determined by a content adaptive training process. The processing circuitry determines the DNN for a first boundary region comprising a subset of samples in the reconstructed blocks based on the first deblocking parameter included in the first deblocking information. The processing circuitry deblocks the first boundary region comprising the subset of samples in the reconstructed blocks based on the determined DNN corresponding to the first deblocking parameter.

Content-adaptive online training method and apparatus for deblocking in block-wise image compression

An apparatus to facilitate acceleration of machine learning operations is disclosed. The apparatus comprises at least one processor to perform operations to implement a neural network and accelerator logic to perform communicatively coupled to the processor to perform compute operations for the neural network.

Machine learning accelerator mechanism

Systems and methods for deep neural network (DNN)-based cross component prediction are provided. A method includes inputting a reconstructed luma block of an image or video into a DNN; and predicting, by the DNN, a reconstructed chroma block of the image or video based on the reconstructed luma block that is input. Luma and chroma reference information and side information may also be input into the DNN to predict the reconstructed chroma block. The various inputs may also be generated using processes such as downsampling and transformation.

DNN-based cross component prediction

Video processing with a multi-quality loop filter using a multi-task neural network is performed by at least one processor and includes generating a first set of masked weight parameters, based on an input and a plurality of quantization parameter values with a corresponding first set of masks and first plurality of weight parameters, for a first set of shared neural network layers, selecting a second set of task specific neural network layers for the plurality of quantization parameter values with a second plurality of weight parameters, based on the plurality of quantization parameter values, computing an inference output, based on the first set of masked weight parameters and the second plurality of weight parameters, and outputting the computed inference output as an enhanced result.

Multi-task neural network by micro-structured parameter sharing for multi-quality loop filter

A method and apparatus for adaptive neural image compression with smooth quality control by meta-learning includes receiving an input image and a target quality control parameter; generating quality-adaptive weight parameters of an encoding neural network using shared encoding parameters and adaptive encoding parameters; and encoding the input image, based on the target quality control parameter, using the encoding neural network with the quality-adaptive weight parameters, to generate a compressed representation.

Adaptive neural image compression with smooth quality control by meta-learning

A method, computer program, and computer system is provided for compressing a neural network model. A multi-dimensional tensor corresponding to a set of weight coefficients associated with a neural network is reshaped. A subset of weight coefficients is identified from among the set of weight coefficients. A model of the neural network is compressed based on the identified subset of weight coefficients.

Neural network model compression with quantizability regularization

Neural image compression using substitutional quality factor learning in a latent space, including receiving a compressed bitstream and a target quality factor indicating a target compression quality, calculating a decoded latent representation of the compressed bitstream, and calculating a reconstructed image based on the decoded latent representation of the compressed bitstream and the target quality factor, computing a shared feature based on a network forward computation using shared decoding parameters (SDP) of one or more layers of a convolutional neural network, computing estimated adaptive decoding parameters (ADP) for the one or more layers of the convolutional neural network based on the shared feature, the adaptive decoding parameters, and the target quality factor, and computing an output tensor based on the estimated ADP in the one or more layers of the convolutional neural network and the shared feature.

Substitutional quality factor learning in the latent space for neural image compression

Content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, is provided, including receiving an input image, to an E2E NIC framework, including one or more blocks, preprocessing a first neural network of the E2E NIC framework, based on the one or more blocks, computing updated parameters using the preprocessed first neural network, encoding the one or more blocks and the updated parameters, updating the first neural network based on the encoded updated parameters, and generating a compressed representation of the encoded one or more blocks using the updated first neural network.

System, method, and computer program for content adaptive online training for multiple blocks in neural image compression

Neural network based substitutional end-to-end (E2E) image compression (NIC) being performed by at least one processor and includes receiving an input image to an E2E NIC framework, determining a step size of the input image indicating a learning rate of a training model, determining a substitute image based on the training model, encoding the substitute image in lieu of the input image to generate a bitstream, and mapping the substitute image to the bitstream to generate a compressed representation. Further, step size may be determined by a scheduler and change throughout the training of the training model. The image may also be split into patches for which a scheduler is assigned for each patch and each patch is encoded instead of the entire input image.

Method and apparatus for dynamic learning rates of substitution in neural image compression

A method of unification based coding for neural network model compression is performed by at least one processor and includes receiving a layer uniform flag indicating whether a quantized weight of an input neural network is encoded using a uniform coding method, and determining whether the quantized weight is encoded using the uniform coding method, based on the received layer uniform flag. The method further includes, based on the quantized weight being determined to be encoded using the uniform coding method, encoding the quantized weight, using the uniform coding method, and based on the quantized weight being determined to not be encoded using the uniform coding method, encoding the quantized weight, using a non-uniform coding method.

Method and apparatus for unification based coding for neural network model compression

A pruning method of neural network based video coding of a current block of a picture of a video sequence is performed by at least one processor and includes categorizing parameters of a neural network into groups, setting a first index to indicate that a first group of the groups is to be pruned, and a second index to indicate that a second group of the groups is not to be pruned, and transmitting, to a decoder, the set first index and the set second index. Based on the transmitted first index and the transmitted second index, the current block is processed using the parameters of which the first group of the groups is pruned.

Pruning methods for neural network based video coding

Systems and methods for encoding and decoding neural network data is provided. A method includes: obtaining an independent neural network with a topology; encoding the independent neural network with the topology such as to obtain a neural network representation (NNR) bitstream; and sending the NNR bitstream to a decoder, wherein the NNR bitstream includes a group of NNR units (GON) that represents the independent neural network with the topology, and the GON includes an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit, and an NNR compressed data unit.

Techniques for parameter set and header design for compressed neural network representation

Neural network based substitutional end-to-end (E2E) image compression (NIC) being performed by at least one processor and includes receiving an input image to an E2E NIC framework, determining a substitute image based on a training model of the E2E NIC framework, encoding the substitute image to generate a bitstream, mapping the substitute image to the bitstream to generate a compressed representation of the input image. Further, the input may be partitioned into blocks for which a substitute representation is determined for each block and each block is encoded instead of the entire substitute image.

Method and apparatus for multi-learning rates of substitution in neural image compression

A method, computer program, and computer system is provided for video encoding and decoding. Video data including a current picture is received. A virtual reference frame is generated for the current picture based on hierarchical level associated with the current picture and a nearest decoded picture. The video data is decoded based on the generated reference frame.

Growing community of inventors

Palo Alto, CA, United States of America

Wei Wang