Tencent America LLC

Renesas Electronics America Inc.

Wei Wang

Shan Liu

Wei Jiang

Xiaozhong Xu

Sheng Lin

Ding Ding

Byeongdoo Choi

James Wang

Zeqiang Li

Jankin Hu

Liqiang Wang

Shubing Zhai

Stephan Wenger

Systems and methods for deep neural network (DNN)-based cross component prediction are provided. A method includes inputting a reconstructed luma block of an image or video into a DNN; and predicting, by the DNN, a reconstructed chroma block of the image or video based on the reconstructed luma block that is input. Luma and chroma reference information and side information may also be input into the DNN to predict the reconstructed chroma block. The various inputs may also be generated using processes such as downsampling and transformation.

DNN-based cross component prediction

Video processing with a multi-quality loop filter using a multi-task neural network is performed by at least one processor and includes generating a first set of masked weight parameters, based on an input and a plurality of quantization parameter values with a corresponding first set of masks and first plurality of weight parameters, for a first set of shared neural network layers, selecting a second set of task specific neural network layers for the plurality of quantization parameter values with a second plurality of weight parameters, based on the plurality of quantization parameter values, computing an inference output, based on the first set of masked weight parameters and the second plurality of weight parameters, and outputting the computed inference output as an enhanced result.

Multi-task neural network by micro-structured parameter sharing for multi-quality loop filter

A method and apparatus for adaptive neural image compression with smooth quality control by meta-learning includes receiving an input image and a target quality control parameter; generating quality-adaptive weight parameters of an encoding neural network using shared encoding parameters and adaptive encoding parameters; and encoding the input image, based on the target quality control parameter, using the encoding neural network with the quality-adaptive weight parameters, to generate a compressed representation.

Adaptive neural image compression with smooth quality control by meta-learning

A method, computer program, and computer system is provided for compressing a neural network model. A multi-dimensional tensor corresponding to a set of weight coefficients associated with a neural network is reshaped. A subset of weight coefficients is identified from among the set of weight coefficients. A model of the neural network is compressed based on the identified subset of weight coefficients.

Neural network model compression with quantizability regularization

Neural image compression using substitutional quality factor learning in a latent space, including receiving a compressed bitstream and a target quality factor indicating a target compression quality, calculating a decoded latent representation of the compressed bitstream, and calculating a reconstructed image based on the decoded latent representation of the compressed bitstream and the target quality factor, computing a shared feature based on a network forward computation using shared decoding parameters (SDP) of one or more layers of a convolutional neural network, computing estimated adaptive decoding parameters (ADP) for the one or more layers of the convolutional neural network based on the shared feature, the adaptive decoding parameters, and the target quality factor, and computing an output tensor based on the estimated ADP in the one or more layers of the convolutional neural network and the shared feature.

Substitutional quality factor learning in the latent space for neural image compression

Content-adaptive online training for end-to-end (E2E) neural image compression (NIC) using a neural network performed by at least one processor, is provided, including receiving an input image, to an E2E NIC framework, including one or more blocks, preprocessing a first neural network of the E2E NIC framework, based on the one or more blocks, computing updated parameters using the preprocessed first neural network, encoding the one or more blocks and the updated parameters, updating the first neural network based on the encoded updated parameters, and generating a compressed representation of the encoded one or more blocks using the updated first neural network.

System, method, and computer program for content adaptive online training for multiple blocks in neural image compression

Neural network based substitutional end-to-end (E2E) image compression (NIC) being performed by at least one processor and includes receiving an input image to an E2E NIC framework, determining a step size of the input image indicating a learning rate of a training model, determining a substitute image based on the training model, encoding the substitute image in lieu of the input image to generate a bitstream, and mapping the substitute image to the bitstream to generate a compressed representation. Further, step size may be determined by a scheduler and change throughout the training of the training model. The image may also be split into patches for which a scheduler is assigned for each patch and each patch is encoded instead of the entire input image.

Method and apparatus for dynamic learning rates of substitution in neural image compression

A method of unification based coding for neural network model compression is performed by at least one processor and includes receiving a layer uniform flag indicating whether a quantized weight of an input neural network is encoded using a uniform coding method, and determining whether the quantized weight is encoded using the uniform coding method, based on the received layer uniform flag. The method further includes, based on the quantized weight being determined to be encoded using the uniform coding method, encoding the quantized weight, using the uniform coding method, and based on the quantized weight being determined to not be encoded using the uniform coding method, encoding the quantized weight, using a non-uniform coding method.

Method and apparatus for unification based coding for neural network model compression

A pruning method of neural network based video coding of a current block of a picture of a video sequence is performed by at least one processor and includes categorizing parameters of a neural network into groups, setting a first index to indicate that a first group of the groups is to be pruned, and a second index to indicate that a second group of the groups is not to be pruned, and transmitting, to a decoder, the set first index and the set second index. Based on the transmitted first index and the transmitted second index, the current block is processed using the parameters of which the first group of the groups is pruned.

Pruning methods for neural network based video coding

Systems and methods for encoding and decoding neural network data is provided. A method includes: obtaining an independent neural network with a topology; encoding the independent neural network with the topology such as to obtain a neural network representation (NNR) bitstream; and sending the NNR bitstream to a decoder, wherein the NNR bitstream includes a group of NNR units (GON) that represents the independent neural network with the topology, and the GON includes an NNR model parameter set unit, an NNR layer parameter set unit, an NNR topology unit, an NNR quantization unit, and an NNR compressed data unit.

Techniques for parameter set and header design for compressed neural network representation

Neural network based substitutional end-to-end (E2E) image compression (NIC) being performed by at least one processor and includes receiving an input image to an E2E NIC framework, determining a substitute image based on a training model of the E2E NIC framework, encoding the substitute image to generate a bitstream, mapping the substitute image to the bitstream to generate a compressed representation of the input image. Further, the input may be partitioned into blocks for which a substitute representation is determined for each block and each block is encoded instead of the entire substitute image.

Method and apparatus for multi-learning rates of substitution in neural image compression

A method, computer program, and computer system is provided for video encoding and decoding. Video data including a current picture is received. A virtual reference frame is generated for the current picture based on hierarchical level associated with the current picture and a nearest decoded picture. The video data is decoded based on the generated reference frame.

Utilizing hierarchical structure for neural network based tools in video coding

Systems and methods for block-wise entropy coding methods in neural image compression is provided. A method includes: receiving a bitstream that includes an image; partitioning the image into a plurality of blocks; compressing each of the plurality of blocks by a neural network-based encoder; obtaining compressed features by obtaining a compressed feature for each block from among the plurality of blocks in the image; processing the compressed features by an entropy encoder to generate a first compressed bitstream; obtaining a plurality of reshaped compressed features by concatenating the compressed features; processing the plurality of reshaped compressed features by the entropy encoder to generate a second compressed bitstream; and encoding the bitstream including the image based on the second compressed bitstream.

Block-wise entropy coding method in neural image compression

According to embodiments, a method of neural network optimized matrix-matrix multiplication (NNMM) is performed by at least one processor and includes determining a first matrix of input coefficients, and determining a second matrix of weight coefficients of a neural network that is trained so that the second matrix has a predetermined uniform pattern, the predetermined uniform pattern having at least two of the weight coefficients that have a same value. The method further includes performing multiplication of the determined first matrix and the determined second matrix to determine output coefficients.

Method and apparatus for neural network optimized matrix-matrix multiplication (NNMM)

Systems and methods for controlling data transaction between master and slave devices are described. A master device can be connected to multiple slave devices that can operate under one of a first, a second, and a third operation modes. The first operation mode can cause the master device to perform data transactions with the multiple slave devices via a network element and the multiple slave devices can be connected to one another via the network element. The second operation mode can disconnect the master device from the multiple slave devices, and multiple agents connected to the multiple slave devices can fulfill the data transactions. The third operation mode can cause the master device to perform data transactions with a first subset of the multiple slave devices via the network element, and can cause the master device to be disconnected from a second subset of the multiple slave devices.

Growing community of inventors

Palo Alto, CA, United States of America

Wei Wang