City University of Hong Kong

Tfi Digital Media Limited

Centre for Intelligent Multidimensional Data Analysis Limited

Centre for Intelligent Multidimensaional Data Analysis Limited

Sam Tak Wu Kwong

Shiqi Wang

Yun Zhang

Hon Wah Wong

Chi Keung Fong

Yiu Fai Yuen

Zhangkai Ni

Hok Kwan Cheung

Hon Tung Luk

Long Xu

Mingliang Zhou

Yue Liu

Linwei Zhu

Xuekai Wei

Dongjie Ye

Meng Wang

Jing Liao

Wei Gao

Yi Chen

Zhenhao Sun

Jia Zhang

Jialu Huang

Yunhao Mao

Haifeng Guo

Chongyi Li

Xinju Wu

A method for adaptive reconstruction of a compressively sensed data. The method contains the steps of receiving sensed data; conducting an initial reconstruction to the sensed data to obtain a plurality of first reconstruction patches; by a reconstruction module, conducting a progressive reconstruction to the sensed data to obtain a plurality of second reconstruction patches; summing the plurality of second reconstruction patches with the a plurality of first reconstruction patches to obtain final patches; and merging the final patches to obtain a reconstructed data. The progressive reconstruction further contains concatenating transformer features and convolution features to obtain the second reconstruction patches. The invention provides a hybrid network for adaptive sampling and reconstruction of CS, which integrates the advantages of leveraging both detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning.

Convolution and transformer based compressive sensing

A computer-implemented method for facilitating encoding of video data. The method includes performing an operation to determine prediction residuals associated with a unit of the video data, and, processing, using a neural network arrangement, the prediction residuals associated with the unit of the video data to determine model parameters associated with a rate-distortion model for the unit of the video data. The model parameters are arranged to facilitate encoding of at least the unit of the video data. The method can be applied to multiple ones of such unit of the video data.

Facilitating encoding of video data using neural network

There is provided a computer-implemented method for learned video compression, which includes processing a current frame (x) and previously decoded frame ({circumflex over (x)}) of a video data using a motion estimation model to estimate a motion vector (v) for every pixel, compressing the motion vector (v) and reconstructing the motion vector (v) to a reconstructed motion vector ({circumflex over (v)}), applying an enhanced context mining (ECM) model to obtain enhanced context ({umlaut over (C)}) from the reconstructed motion vector ({circumflex over (v)}) and previously decoded frame feature (x̆), compressing the current frame (x) with the assistance of the enhanced context ({umlaut over (C)}) to obtain a reconstructed frame ({circumflex over (x)}'), and providing the reconstructed frame ({circumflex over (x)}′) to a post-enhancement backend network to obtain a high-resolution frame ({circumflex over (x)}).

Method and system for learned video compression

A method for learning-based point cloud geometry compression includes: given a source point cloud, regressing an aligned mesh that is driven by a set of parameters from a deformable template mesh, quantizing the set of parameters into a parameter bitstream, generating an aligned point cloud from the quantized parameters by mesh manipulation and mesh-to-point-cloud conversion, extracting features from both the source point cloud and the aligned point cloud based on sparse tensors including coordinates and features, the coordinates being encoded into a coordinate bitstream, warping the features of the aligned point cloud onto the coordinates of the source point cloud, obtaining residual features through feature subtraction, processing the residual features using an entropy model into a residual feature bitstream, and obtaining a reconstructed point cloud by processing the parameter bitstream, the coordinate bitstream and the residual feature bitstream.

Point cloud geometry compression

A system and a method for processing an image. The system comprises an image gateway arranged to receive an input image showing a scene composed by a combination of a plurality of image portions of the input image, wherein one or more of the plurality of image portions is associated with an exposure level deviated from an optimal exposure level; and an enhancement engine arranged to process the input image by applying an exposure/image relationship to the input image, wherein the exposure/image relationship is arranged to adjust the exposure level of each of the plurality of image portions towards the optimal exposure level; and to generate an enhanced image showing a visual representation of the scene composed by a combination of the plurality of image portions of the input image with an adjusted exposure level.

System and a method for processing an image

A method of providing an image to image translation using a style based generator architecture for generative adversarial networks. The method includes the step of applying a feature alignment loss function defined by:Loss=Σ1(ϕ(),ϕ())wherein Gand Gare respective generators of a source model and a target model, ϕ(·)is the extraction of the ith layer of features and wrepresents the weight of Lloss at the ith layer, the Lloss weight wbeing reduced as layers decrease such that Lloss at higher layers has a higher weight than the Lloss at lower layers.

Image to image translation method

Systems and methods for genome sequence compression and decompression are provided. The method for compression encoding of a genome sequence includes partitioning a genome sequence into a plurality of Group of Bases (GoBs) and processing each of the plurality of GoBs independently to encode the genome sequence into a bit stream. Processing each of the plurality of GoBs includes dividing each of the plurality of GOBs into a first part and a second part, the first part including an initial context part and the second part including a learning-based inference part. The processing each of the plurality of GoBs further includes encoding the first part in accordance with a Markov model, encoding the second part in accordance with a learning-based model, and encoding the encoded first part and the encoded second part into the bit stream with an arithmetic encoder. The learning-based model may include Long and Short-Term Memory (LSTM)-based neural networks.

Method and systems for genome sequence compression

A method for processing a stream of images including the steps of obtaining coding information from the stream of images to determine one or more bitrate/distortion models representative of the bitrate/distortion relationship of the stream of images, determining a set of coding parameters arranged for use to encode a stream of images with the one or more bitrate/distortion models, reformulating the bitrate/distortion relationship into a decoupled relationship arranged to be applied to a subset of the stream of images, and using the decoupled relationship and the set of coding parameters to generate an adaptive quantization parameter for encoding the stream of images with the bitrate/distortion relationship.

System and method for processing a stream of images

A method for streaming a video. The method includes determining a total bitrate for a segment of a video to be received and streamed; predicting a viewpoint of a user for the segment; and determining bitrates for one or more tiles in the segment based on the determined total bitrate and the predicted viewpoint.

Video streaming method and system

System and method for adaptively streaming a video. The method includes obtaining a first segment of a video file with a first bitrate and video quality level, and, after obtaining the first segment of the video file, determining, using a controller with a trained reinforcement-learning-based adaptive streaming model, a second bitrate and video quality level of a second segment of the video file to be obtained. The method also includes obtaining the second segment of the video file with the second determined bitrate and video quality level. The method may be repeated for different segments of the video file so as to adaptively stream the video represented by the video file.

System and method for adaptively streaming video

A system and method for processing an image including the steps of receiving an input image having a plurality of pixels, wherein each of the plurality of pixels have one or more pixel characteristics; and processing the input image to generate an enhanced image by applying a pixel/image relationship to each of the plurality of pixels of the input image, wherein the pixel/image relationship is arranged to adjust the one or more pixel characteristics of each of the plurality of pixels of the input image.

System and method for processing an image

A computer-implemented method and related system for determining a quality of a synthesized video file. The method includes processing a reference video file and a synthesized video file associated with the reference video file to compare the original video file and the synthesized video file. The method also includes determining an extent of flicker distortion of the synthesized video file based on the processing.

Video quality determination system and method

Systems and methods which provide Generative Adversarial Network (GAN) based intra prediction for video coding are described. GAN based intra prediction as implemented according to embodiments may be modeled as an inpainting task. For example, intra prediction may be formulated as a learning based inpainting task, wherein a latent variable is designed to control different generation modes. GAN based intra prediction provided according to embodiments of the invention may be implemented alone or in combination with one or more other video compression technique, such as a direction intra prediction technique. The intra prediction module of such a HEVC encoder/decoder may be redesigned to also apply GAN based inpainting in intra prediction, wherein Rate-Distortion Optimization (RDO) may be performed to select the best intra prediction mode between the intra prediction approaches.

Generative adversarial network based intra prediction for video coding

Systems and methods which provide Convolutional Neural Network (CNN) based synthesized view quality enhancement for video coding are described. Embodiments may comprise an encoder configured for CNN based synthesized view quality enhancement configured to provide improved coding efficiency while maintaining synthesized view quality. Additionally or alternatively, embodiments may comprise a virtual viewpoint generator configured for CNN based synthesized view quality enhancement configured provide post-processing of the synthesized view at the decoder side to reduce the artifacts. CNN based synthesized view quality enhancement may, for example, be provided for 3D video coding to improve its coding efficiency, which can be utilized in 3D scenarios, such as 3DTV.

Growing community of inventors

Hong Kong, China

Sam Tak Wu Kwong