Google Inc.

Claudionor Coelho

Aki Kuusela

Shan Li

Dake He

Joseph Young

Sean Purser-Haskell

Hao Zhuang

In Suk Chong

Daniele Moro

Stan Vitvitskyy

This document describes systems and techniques for encoding a video frame using different compression ratios or compression algorithms for text blocks and non-text blocks. The described systems and techniques can determine, using a machine-learned model, which blocks of a frame include and do not include text. The described systems and techniques can then use a different compression ratio or compression algorithm for text blocks than the compression ratio or compression algorithm used for non-text blocks. For example, the systems and techniques can encode the text blocks using a first compression ratio that results in higher video quality than a second compression ratio used on at least some non-text blocks. In this way, the described systems and techniques can improve text legibility in a video file without significantly increasing the bandwidth requirements to transmit the video file to remote computing devices.

Encoding a video frame using different compression ratios for text blocks and non-text blocks

Ultra light models and decision fusion for increasing the speed of intra-prediction are described. Using a machine-learning (ML) model, an ML intra-prediction mode is obtained. A most-probable intra-prediction mode is obtained from amongst available intra-prediction modes for encoding the current block. As an encoding intra-prediction mode, one of the ML intra-prediction mode or the most-probable intra-prediction mode is selected, and the encoding intra-prediction mode is encoded in a compressed bitstream. A current block is encoded using the encoding intra-prediction mode. Selection of the encoding intra-prediction mode is based on relative reliabilities of the ML intra-prediction mode and the most-probable intra-prediction mode.

Ultra light models and decision fusion for fast video coding

An apparatus for encoding an image block includes a processor that presents, to a machine-learning model, the image block, obtains the partition decision for encoding the image block from the model, and encodes the image block using the partition decision. The model is trained to output a partition decision for encoding the image block by using training data for a plurality of training blocks as input, the training data including for a training block, partition decisions for encoding the training block, and, for each partition decision, a rate-distortion value resulting from encoding the training block using the partition decision. The model is trained using a loss function combining a partition loss function based upon a relationship between the partition decisions and respective predicted partitions, and a rate-distortion cost loss function based upon a relationship between the rate-distortion values and respective predicted rate-distortion values.

Using rate distortion cost as a loss function for deep learning

Encoding an image block using a quantization parameter includes presenting, to an encoder that includes a machine-learning model, the image block and a value derived from the quantization parameter, where the value is a result of a non-linear function using the quantization parameter as input, where the non-linear function relates to a second function used to calculate, using the quantization parameter, a Lagrange multiplier that is used in a rate-distortion calculation, and where the machine-learning model is trained to output mode decision parameters for encoding the image block; obtaining the mode decision parameters from the encoder; and encoding, in a compressed bitstream, the image block using the mode decision parameters.

Efficient use of quantization parameters in machine-learning models for video coding

An apparatus for encoding a block of a picture includes a convolutional neural network (CNN) for determining a block partitioning of the block, the block having an N&#xd7;N size and a smallest partition determined by the CNN being of size S&#xd7;S. The CNN includes feature extraction layers; a concatenation layer that receives, from the feature extraction layers, first feature maps of the block, where each first feature map of the first feature maps is of the smallest possible partition size S&#xd7;S of the block; and at least one classifier that is configured to infer partition decisions for sub-blocks of size (&#x3b1;S)&#xd7;(&#x3b1;S) of the block, where &#x3b1; is a power of 2.

Receptive-field-conforming convolutional models for video coding

Convolutional neural networks (CNN) that determine a mode decision (e.g., block partitioning) for encoding a block include feature extraction layers and multiple classifiers. A non-overlapping convolution operation is performed at a feature extraction layer by setting a stride value equal to a kernel size. The block has a N&#xd7;N size, and a smallest partition output for the block has a S&#xd7;S size. Classification layers of each classifier receive feature maps having a feature dimension. An initial classification layer receives the feature maps as an output of a final feature extraction layer. Each classifier infers partition decisions for sub-blocks of size (&#x3b1;S)&#xd7;(&#x3b1;S) of the block, wherein &#x3b1; is a power of 2 and &#x3b1;=2, . . . , N/S, by applying, at some successive classification layers, a 1&#xd7;1 kernel to reduce respective feature dimensions; and outputting by a last layer of the classification layers an output corresponding to a N/(&#x3b1;S)&#xd7;N/(&#x3b1;S)&#xd7;1 output map.

Receptive-field-conforming convolution models for video coding

A convolutional neural network (CNN) for determining a partitioning of a block is disclosed. The block is of size N&#xd7;N and a smallest partition is of size S&#xd7;S. The CNN includes feature extraction layers; a concatenation layer that receives, from the feature extraction layers, first feature maps of the block, where each first feature map is of size S&#xd7;S; and classifiers. Each classifier includes classification layers, each classification layer receives second feature maps having a respective feature dimension. Each classifier is configured to infer partition decisions for sub-blocks of size (&#x3b1;S)&#xd7;(&#x3b1;S) of the block, wherein &#x3b1; is a power of 2 and &#x3b1;=2, . . . , N/S, by: applying, at some of successive classification layers of the classification layers, a kernel of size 1&#xd7;1 to reduce the respective feature dimension in half; and outputting by a last layer of the classification layers an output corresponding to a N/(&#x3b1;S)&#xd7;N/(&#x3b1;S)&#xd7;1 output map.

A method for encoding an image block includes presenting, to a machine-learning model, the image block and a first value corresponding to a first quantization parameter; obtaining first mode decision parameters from the machine-learning model; and encoding the image block using the first mode decision parameters. The first value results from a non-linear function using the first quantization parameter as input. The machine-learning model is trained to output mode decision parameters by using training data. Each training datum includes a training block that is encoded by a second encoder, second mode decision parameters used by the second encoder for encoding the training block, and a second value corresponding to a second quantization parameter. The second encoder used the second quantization parameter for encoding the training block and the second value results from the non-linear function using the second quantization parameter as input.

Growing community of inventors

Redwood City, CA, United States of America

Claudionor Coelho