City University of Hong Kong

Beijing Bytedance Network Technology Co., Ltd.

Bytedance Inc.

Alibaba Damo (hangzhou) Technology Co., Ltd.

Bytedance (hk) Limited

Lemon Inc.

Centre for Intelligent Multidimensional Data Analysis Limited

Centre for Intelligent Multidimensaional Data Analysis Limited

Shiqi Wang

Sam Tak Wu Kwong

Li Feng Zhang

Yan Ye

Meng Wang

Kai Zhang

Zhao Wang

Junru Li

Hongbin Liu

Yue Wang

Shurun Wang

Zhangkai Ni

Yue Liu

Bolin Chen

Jizheng Xu

Siwei Ma

Yue Li

Dongjie Ye

Kede Ma

Keyan Ding

Yi Chen

Chenqi Kong

Kexin Zheng

Haoliang Li

Yunhao Mao

Haifeng Guo

Xinju Wu

Zhenhao Sun

Methods and apparatuses are provided for compressing video data based on keypoint features. An exemplary video compression method includes: receiving a video sequence; encoding one or more pictures of the video sequence; and generating a bitstream; wherein the encoding includes: representing a first picture by a first set of keypoints and a second set of keypoints, the second set comprising less keypoints than the first set; and compressing the video sequence based on the first set and second set of keypoints.

Keypoints based video compression

Methods and apparatuses are provided for processing video data. An exemplary method includes: decompressing a compressed frame to generate a key frame representing a face; generating, for the key frame, a first set of parameters associated with a 3-dimensional (3D) face representation of the face; reconstructing, for each of one or more inter frames, a second set of parameters associated with a 3D face representation of the face according to compressed inter-predicted residuals of the second set of parameters; and generating a video comprising the face based on the key frame, the first set of parameters, and the second set of parameters.

Method and apparatus for talking face video compression

The present disclosure provides pre-analysis based methods for adaptively compressing image data consumed by machine vision tasks. An exemplary method includes: receiving a video sequence; encoding one or more input pictures associated with the video sequence; and generating a bitstream, wherein the encoding includes: performing instance segmentation of an input picture, to generate one or more segment masks; combining the one or more segment masks to generate a merged mask; extracting, from the input picture, a region comprising the merged mask; and compressing image data representing the extracted region.

Pre-analysis based image compression methods

An image data encoding method is provided. The image data encoding method includes receiving an input image and a quantization parameter (QP); performing pre-analysis of the input image based on the QP to obtain input image data for machine version; determining at least one of a temporal redundancy or a spatial redundancy of the input image data; determining, based on the at least one of temporal redundancy or spatial redundancy, whether to perform at least one of temporal down-sampling or spatial down-sampling; performing down-sampling of the input image data according to the determination, wherein the down-sampling includes at least one of temporal down-sampling or spatial down-sampling; and encoding the down-sampled image data based on the QP.

Methods and non-transitory computer readable storage medium for pre-analysis based resampling compression for machine vision

A computer-implemented method for facilitating encoding of video data. The method includes performing an operation to determine prediction residuals associated with a unit of the video data, and, processing, using a neural network arrangement, the prediction residuals associated with the unit of the video data to determine model parameters associated with a rate-distortion model for the unit of the video data. The model parameters are arranged to facilitate encoding of at least the unit of the video data. The method can be applied to multiple ones of such unit of the video data.

Facilitating encoding of video data using neural network

There is provided a computer-implemented method for learned video compression, which includes processing a current frame (x) and previously decoded frame ({circumflex over (x)}) of a video data using a motion estimation model to estimate a motion vector (v) for every pixel, compressing the motion vector (v) and reconstructing the motion vector (v) to a reconstructed motion vector ({circumflex over (v)}), applying an enhanced context mining (ECM) model to obtain enhanced context ({umlaut over (C)}) from the reconstructed motion vector ({circumflex over (v)}) and previously decoded frame feature (x̆), compressing the current frame (x) with the assistance of the enhanced context ({umlaut over (C)}) to obtain a reconstructed frame ({circumflex over (x)}'), and providing the reconstructed frame ({circumflex over (x)}′) to a post-enhancement backend network to obtain a high-resolution frame ({circumflex over (x)}).

Method and system for learned video compression

A video processing method includes performing a conversion between a video including a video unit and a coded representation of the video, where, after the video unit is encoded or decoded with an intra prediction mode, one or more frequence tables and/or one or more sorted intra prediction mode (IPM) tables are selectively updated according to a rule, where the one or more frequence tables include information about frequence of the intra prediction mode used for processing the video unit in the conversion, where the frequence indicates an occurrence of the intra prediction mode used for the conversion, and where the one or more sorted IPM tables indicate the intra prediction mode used in the processing.

Updating for counter-based intra prediction mode

A method for learning-based point cloud geometry compression includes: given a source point cloud, regressing an aligned mesh that is driven by a set of parameters from a deformable template mesh, quantizing the set of parameters into a parameter bitstream, generating an aligned point cloud from the quantized parameters by mesh manipulation and mesh-to-point-cloud conversion, extracting features from both the source point cloud and the aligned point cloud based on sparse tensors including coordinates and features, the coordinates being encoded into a coordinate bitstream, warping the features of the aligned point cloud onto the coordinates of the source point cloud, obtaining residual features through feature subtraction, processing the residual features using an entropy model into a residual feature bitstream, and obtaining a reconstructed point cloud by processing the parameter bitstream, the coordinate bitstream and the residual feature bitstream.

Point cloud geometry compression

Two-dimensional face presentation attacks are one of most notorious and pervasive face spoofing types, causing security issues to facial authentication systems. To tackle these issues, a cost-effective face anti-spoofing (FAS) system based on acoustic modality, named as Echo-FAS, is devised, which employs a crafted acoustic signal to probe the presented face. First, a large-scale, high-diversity, acoustic-based FAS database, named as Echo-Spoof, is built. Based upon Echo-Spoof, we design a two-branch framework combining global and local frequency features of the presented face to distinguish live vs. spoofing faces. Echo-FAS has the following merits: (1) it only needs one speaker and one microphone; (2) it can capture three-dimensional geometrical information of the presented face and achieve a remarkable FAS performance; and (3) it can be handily allied with RGB-based FAS models to mitigate the overfitting problem in the RGB modality and make the FAS model more accurate and robust.

Acoustic-based face anti-spoofing system and method

Embodiments of the present disclosure provide a solution for image/video super resolution. A method for image processing is proposed. The method comprises: receiving a first image with a first resolution and at least one reference image associated with the first image, the first image and the at least one reference image being associated with a same video; determining a difference between the first image and the at least one reference image; and generating a second image with a second resolution based on the difference, the first image and the at least one reference image, the second resolution being higher than the first resolution.

Image/video super resolution

The present application provides methods, devices and computer readable media for intrinsic popularity evaluation and content compression based thereon. In an embodiment, there is provided a method of intrinsic popularity evaluation. The method comprises: receiving an image from a social network; and determining an intrinsic popularity score for the image using a deep neural network (DNN) based intrinsic popularity assessment model.

Method, device and computer readable medium for intrinsic popularity evaluation and content compression based thereon

A system and a method for processing an image. The system comprises an image gateway arranged to receive an input image showing a scene composed by a combination of a plurality of image portions of the input image, wherein one or more of the plurality of image portions is associated with an exposure level deviated from an optimal exposure level; and an enhancement engine arranged to process the input image by applying an exposure/image relationship to the input image, wherein the exposure/image relationship is arranged to adjust the exposure level of each of the plurality of image portions towards the optimal exposure level; and to generate an enhanced image showing a visual representation of the scene composed by a combination of the plurality of image portions of the input image with an adjusted exposure level.

System and a method for processing an image

A method implemented by a video coding apparatus includes applying a neural network (NN) filter to an unfiltered sample of a video unit to generate a filtered sample. The NN filter is applied based on a syntax element of the video unit. The method also includes converting between a video media file and a bitstream based on the filtered sample that was generated.

Unified neural network in-loop filter signaling

A video processing method includes performing a conversion between a video comprising multiple video units and a coded representation of the video in which the multiple video units are processed in an order, where, when a video unit of the multiple video units is processed in the conversion, one or more frequence tables and/or one or more sorted intra prediction mode (IPM) tables are reset or initialized, where the one or more frequence tables include information about frequence of one or more intra prediction modes used for processing previous video units in the multiple video units, where the previous video units precede in time the video unit, where the frequence indicates an occurrence of the one or more intra prediction modes used for the conversion, and where the one or more sorted IPM tables indicate the one or more intra prediction modes used in the processing.

Initialization for counter-based intra prediction mode

Systems and methods for genome sequence compression and decompression are provided. The method for compression encoding of a genome sequence includes partitioning a genome sequence into a plurality of Group of Bases (GoBs) and processing each of the plurality of GoBs independently to encode the genome sequence into a bit stream. Processing each of the plurality of GoBs includes dividing each of the plurality of GOBs into a first part and a second part, the first part including an initial context part and the second part including a learning-based inference part. The processing each of the plurality of GoBs further includes encoding the first part in accordance with a Markov model, encoding the second part in accordance with a learning-based model, and encoding the encoded first part and the encoded second part into the bit stream with an arithmetic encoder. The learning-based model may include Long and Short-Term Memory (LSTM)-based neural networks.

Growing community of inventors

Hong Kong, China

Shiqi Wang