Google Inc.

Hewlett-packard Development Company, L.p.

Debargha Mukherjee

Yue Chen

Urvang Joshi

Yaowu Xu

James Bankoski

Chen Wu

Sarah Parker

Jingning Han

Yuxin Liu

Meng Wang

Elliott Karpilovsky

Amir Said

Lester Lu

Vijnan Shastri

Wai-Tian Tan

Lei Yang

Huisheng Wang

Andrew Fergus Simpson

Shunyao Li

Michele Mae Covell

Paul Gordon Wilkins

Jonathan J Huang

Rohit Shankar Singh

Rushabh Ashok Doshi

Ronald Sebastiaan Bultje

Wendy Tobagus

Ken Kengkuan Lin

Mei Chen

Ramin D Zabih

Chen Wang

Sam Kvaalen

Krishna Kumar Gadepalli

Yongzhe Wang

Jim Bankoski

Christophe Chaland

Chuohao Yeo

Emil Keyder

Geraldine Kuo

Aamir Anis

Jianle Chen

Suk Hwan Lim

D Amnon Silverstein

Daniel R Tretter

Ullas Gargi

Niranjan Damera-Venkata

Jonathan Yen

Nelson Liang An Chang

Anil Christopher Kokaram

Damien Kelly

Peyman Milanfar

Andrew Crawford

Hui Su

Nasir D Memon

Mohammed Golam Sarwer

Andrew J Patti

Bowon Lee

Yunqing Wang

Sam Liu

Jiangyang Zhang

Nirmalya Ray

Yuchen Xie

Wei-Ting Lin

Rachel Barker

Obtaining a restored frame from a degraded frame includes obtaining, for a pixel of the degraded frame, magnitude features based on a first window centered at the pixel. A cardinality N of the magnitude features is at least 1. The magnitude features are used to obtain a pixel-adaptive filter. The pixel-adaptive filter is applied to the pixel to obtain a pixel of the restored frame.

Filtering with side-information using contextually-designed filters

Image coding using guided machine learning restoration may include obtaining reconstructed frame data by decoding, obtaining a restored frame by restoring the reconstructed frame, and outputting the restored frame. Obtaining the restored frame may include obtaining a reconstructed block, obtaining guide parameter values, obtaining a restored block, and including the restored block in the restored frame. Obtaining the restored block may include inputting the reconstructed block to an input layer of a trained guided convolutional neural network, wherein the neural network is constrained such that an output layer has a defined cardinality of channels, obtaining, from the output layer, neural network output channel predictions, obtaining a guided neural network prediction as a linear combination of the guide parameter values and the neural network output channel predictions, and generating the restored block using the guided neural network prediction.

Video coding with guided machine learning restoration

Residual coding using vector quantization (VQ) is described. A flag indicating whether a residual block for the current block is encoded using VQ. In response to the flag indicating that the residual block is encoded using VQ, a parameter indicating an entry in a codebook is decoded, and the residual block is decoded using the entry. In response to the flag indicating that the residual block is not encoded using VQ, the residual block is decoded based on a skip flag indicating whether the current block is encoded using transform skip. The current block is reconstructed using the residual block.

Vector quantization for prediction residual coding

Encoding and decoding using warped reference list includes generating a reconstructed frame from an encoded bitstream by, for decoding a current block for the reconstructed frame, obtaining a dynamic reference list, obtaining a warped reference list, decoding a warped reference list index value, obtaining optimal predicted warped model parameters from the warped reference list in accordance with the index value, decoding differential warped model parameters, obtaining, as optimal warped model parameters, a result of adding the optimal predicted warped model parameters and the differential warped model parameters, obtaining predicted block data in accordance with the optimal warped model parameters, decoding residual block data, and obtaining, as decoded block data for the current block, a result of adding the residual block data and the predicted block data.

Warped reference list for warped motion video coding

A portion Y of a degraded frame is restored using a projection operation that uses a first projection parameter α, a second projection parameter β, and at least two guide portions. Restoring the portion Y of the degraded frame includes generating, using first restoration parameters, a first guide portion Yfor the portion Y; generating, using second restoration parameters, a second guide portion Yfor the portion Y; and generating a reconstructed portion Y, wherein the projection operation is based on α(Y−Y)+β(Y−Y).

Restoration for video coding with self-guided filtering and subspace projection

Transform modes are derived for inter-predicted blocks using side information. A prediction residual is generated for a current video block using a reference frame. Side information associated with one or both of the current video block or the reference frame is identified. A trained transform is determined from amongst multiple trained transforms based on the side information, in which each of the trained transforms is determined using individual side information types and combinations of the individual side information types and the side information represents values of one of the individual side information types or one of the combinations of the individual side information types. The prediction residual is transformed according to the trained transform, and data associated with the transformed prediction residual and the side information are encoded to a bitstream.

Inter-prediction mode-dependent transforms for video coding

A method for intra-prediction of a current block includes selecting peripheral pixels of the current block, where the peripheral pixels are used to generate a prediction block for the current block; for each prediction pixel of the prediction block, performing steps including selecting two respective pixels of the peripheral pixels; and calculating the prediction pixel by interpolating at least the two respective pixels; and coding a residual block corresponding to a difference between the current block and the prediction block.

Intra prediction for image and video compression

A method for coding a current block using an intra-prediction mode includes obtaining a focal point, the focal point having coordinates (a, b) in a coordinate system; and generating, using first peripheral pixels and second peripheral pixels, a prediction block for the current block, where the first peripheral pixels form a first peripheral pixel line constituting an x-axis, and where the second peripheral pixels form a second peripheral pixel line constituting a y-axis. Generating the prediction block includes, for each location of the prediction block at a location (i, j) of the prediction block, determining at least one of an x-intercept or a y-intercept; and determining a prediction pixel value for the each location of the prediction block using the at least one of the x-intercept or the y-intercept.

A super-resolution coding mode is described. An encoded image can be decoded from an encoded bitstream stored on a non-transitory computer-readable storage medium. A flag can indicate whether an image was encoded using the super-resolution mode at a first resolution. Responsive to the flag indicating that the image was encoded using the super-resolution mode, bits indicating an amount of scaling of the image are included. The image is decoded from the encoded bitstream to obtain a reconstructed image at the first resolution, and the reconstructed image is upscaled to a second resolution using the amount of scaling to obtain an upscaled reconstructed image. The second resolution is higher than the first resolution. Loop restoration parameters within the bitstream can used for look restoration filtering of the upscaled reconstructed image to obtain a loop restored image at the second resolution.

Super-resolution loop restoration

Generating a compound predictor block for a current block of video includes generating, for the current block, a first predictor block using one of inter-prediction or intra-prediction and generating a second predictor block. The first predictor block includes a first pixel and the second predictor block includes a second pixel that is co-located with the first pixel. A first weight is determined for the first pixel using a difference between a value of the first pixel and a value of the second pixel. A second weight is determined for the second pixel using the first weight. The compound predictor block is generated by combining the first predictor block and the second predictor block. The compound predictor block includes a weighted pixel that is determined using a weighted sum of the first pixel and the second pixel using the first weight and the second weight.

Compound prediction for video coding

Decoding video data includes, for a block encoded using a prediction mode, determining a transform mode for the block using the prediction mode. The transform mode is a first mode when the prediction mode is an inter-prediction mode and is a second mode when the prediction mode is an intra-prediction mode. The first mode is an available first transform type that is a combination of transforms selected from first fixed transforms and first learned transforms that each comprise a respective transformation matrix generated iteratively using blocks predicted using the inter-prediction mode. The second mode is an available second transform type that is a combination of transforms selected from second fixed transforms, which is a proper subset of the first fixed transforms, and a second learned transform comprising a transformation matrix that is generated iteratively using blocks predicted using the intra-prediction mode. Decoding the block uses the prediction and transform modes.

Combination of mode-dependent and fixed transform types in video coding

A device for restoring a degraded frame resulting from reconstruction of a source frame includes a processor that is configured to receive a compressed bitstream. The compressed bitstream includes a first projection parameter α a second projection parameter β, first restoration parameters comprising a first radius value, and second restoration parameters comprising a second radius value. The processor is further configured to restore at least a portion of the degraded frame using a projection operation that uses the first projection parameter α, the second projection parameter β, and at least two guide tiles.

Restoration in video coding using filtering and subspace projection

Improved transforms are used to encode and decode large video and image blocks. During encoding, a prediction residual block having a large size (e.g., larger than 32×32) is generated. The pixel values of the prediction residual block are transformed to produce transform coefficients. After determining that the transform coefficients exceed a threshold cardinality representative of a maximum transform block size (e.g., 32×32), a number of the transform coefficients are discarded such that a remaining number of transform coefficients does not exceed the threshold cardinality. A transform block is then generated using the remaining number. During decoding, after determining that the transform coefficients exceed the threshold cardinality, a number of new coefficients are added to the transform coefficients such that a total number of transform coefficients exceeds the threshold cardinality. The transform coefficients are then inverse transformed into a prediction residual block having a large size.

Transforms for large video and image blocks

A hybrid apparatus for coding a video stream includes a first encoder. The first encoder includes a neural network having at least one hidden layer, and the neural network receives source data from the video stream at a first hidden layer of the at least one hidden layer, receives side information correlated with the source data at the first hidden layer, and generates guided information using the source data and the side information. The first encoder outputs the guided information and the side information for a decoder to reconstruct the source data.

Hybrid motion-compensated neural network with side-information based video coding

Video coding may include generating, by a processor, a decoded frame by decoding a current frame from an encoded bitstream and outputting a reconstructed frame based on the decoded frame. Decoding includes identifying a current encoded block from the current frame, identifying a prediction coding model for the current block, wherein the prediction coding model is a machine learning prediction coding model from a plurality of machine learning prediction coding models, identifying reference values for decoding the current block based on the prediction coding model, obtaining prediction values based on the prediction coding model and the reference values, generating a decoded block corresponding to the current encoded block based on the prediction values, and including the decoded block in the decoded frame.

Image and video coding using machine learning prediction coding models

Growing community of inventors

Cupertino, CA, United States of America

Debargha Mukherjee