Google Inc.

Aki Kuusela

Dake He

Claudionor Coelho

Shan Li

Ville-Mikko Rautio

Paul Leventis

Dov Shimon Zimring

Clinton Smullen

Paul Lalonde

Jani Huoponen

In Suk Chong

Yaowu Xu

Jingning Han

Hui Su

Ching-Han Chiang

Joseph Young

Jean-François Roy

Stanislav Vitvitskyy

Aki Tikkala

Techniques and apparatuses are described for video frame codec architectures. A frame decompressor decompresses compressed frames to produce decompressed frames. A frame decompressor controller arbitrates shared access to the frame decompressor. Multiple cores of an SoC request to receive a decompressed frame from the frame decompressor via the frame decompressor controller. The frame decompressor controller can implement a request queue and can order the servicing of requests based on priority of the requests or requesting cores. The frame decompressor controller can also establish a time-sharing protocol for access by the multiple cores. In some implementations, a video decoder is logically integrated with the frame decompressor and stores portions of a decompressed frame in a video buffer, and a display controller retrieves the portions for display using a synchronization mechanism. In analogous manners, a frame compressor controller can arbitrate shared access to a frame compressor for the multiple cores.

Video frame codec architectures

For a coding block of an image, a luma prediction block is generated, a luma residual block is generated, a quantized luma block is generated after transforming the luma residual block using a luma transform type, and the quantized luma block is entropy encoded. A chroma prediction block is generated, a chroma residual block is generated, an initial chroma transform type for the chroma residual block is determined as the luma transform type, a quantized chroma block is generated using the chroma residual block transformed by a final chroma transform type, and the quantized chroma block is entropy encoded. When the initial chroma transform type is other than a default transform type, the final chroma transform type is the initial chroma transform type or the default transform type, and quantized coefficients of the quantized chroma block depend upon quantized coefficients of the quantized luma block.

Chroma transform type determination

Ultra light models and decision fusion for increasing the speed of intra-prediction are described. Using a machine-learning (ML) model, an ML intra-prediction mode is obtained. A most-probable intra-prediction mode is obtained from amongst available intra-prediction modes for encoding the current block. As an encoding intra-prediction mode, one of the ML intra-prediction mode or the most-probable intra-prediction mode is selected, and the encoding intra-prediction mode is encoded in a compressed bitstream. A current block is encoded using the encoding intra-prediction mode. Selection of the encoding intra-prediction mode is based on relative reliabilities of the ML intra-prediction mode and the most-probable intra-prediction mode.

Ultra light models and decision fusion for fast video coding

An apparatus for encoding an image block includes a processor that presents, to a machine-learning model, the image block, obtains the partition decision for encoding the image block from the model, and encodes the image block using the partition decision. The model is trained to output a partition decision for encoding the image block by using training data for a plurality of training blocks as input, the training data including for a training block, partition decisions for encoding the training block, and, for each partition decision, a rate-distortion value resulting from encoding the training block using the partition decision. The model is trained using a loss function combining a partition loss function based upon a relationship between the partition decisions and respective predicted partitions, and a rate-distortion cost loss function based upon a relationship between the rate-distortion values and respective predicted rate-distortion values.

Using rate distortion cost as a loss function for deep learning

This application is directed to a method of managing processing capability of a server system having one or more processing cores that further include multiple processing slices. Upon receiving requests to initiate online gaming sessions, the server system allocates each processing slice of the processing cores to a subset of the online gaming sessions to be executed thereon. A first processing slice is allocated to a first subset of the online gaming sessions including a first gaming session and a second gaming session. At the first processing slice, a time-sharing processing schedule is determined for the first subset of the online gaming sessions. In accordance with the time-sharing processing schedule, the first and second gaming sessions share a duty cycle of the first processing slice, and are executed dynamically and in parallel according to real-time data processing need of the first and second gaming sessions.

Methods and systems for rendering and encoding content for online interactive gaming sessions

Encoding an image block using a quantization parameter includes presenting, to an encoder that includes a machine-learning model, the image block and a value derived from the quantization parameter, where the value is a result of a non-linear function using the quantization parameter as input, where the non-linear function relates to a second function used to calculate, using the quantization parameter, a Lagrange multiplier that is used in a rate-distortion calculation, and where the machine-learning model is trained to output mode decision parameters for encoding the image block; obtaining the mode decision parameters from the encoder; and encoding, in a compressed bitstream, the image block using the mode decision parameters.

Efficient use of quantization parameters in machine-learning models for video coding

An apparatus for encoding a block of a picture includes a convolutional neural network (CNN) for determining a block partitioning of the block, the block having an N&#xd7;N size and a smallest partition determined by the CNN being of size S&#xd7;S. The CNN includes feature extraction layers; a concatenation layer that receives, from the feature extraction layers, first feature maps of the block, where each first feature map of the first feature maps is of the smallest possible partition size S&#xd7;S of the block; and at least one classifier that is configured to infer partition decisions for sub-blocks of size (&#x3b1;S)&#xd7;(&#x3b1;S) of the block, where &#x3b1; is a power of 2.

Receptive-field-conforming convolutional models for video coding

Asymmetric probability model updating and entropy coding includes using different numbers of bits for storing probabilities of a probability model and for entropy coding symbols using that probability model. The probabilities of a probability model are updated according to values of syntax elements decoded from a bitstream. The probabilities are associated with possible values of the syntax elements and are stored using a first bit precision. Based on the updated probabilities, a second bit precision to use to entropy decode the syntax elements is determined. The second bit precision is less than the first bit precision. The syntax elements are then entropy decoded using the second bit precision, such as to produce quantized transform coefficients, which may be further processed and output to an output video stream. Using the first bit precision to entropy decode the syntax elements results in a lower compression throughput than using the second bit precision.

Asymmetric probability model update and entropy coding precision

Convolutional neural networks (CNN) that determine a mode decision (e.g., block partitioning) for encoding a block include feature extraction layers and multiple classifiers. A non-overlapping convolution operation is performed at a feature extraction layer by setting a stride value equal to a kernel size. The block has a N&#xd7;N size, and a smallest partition output for the block has a S&#xd7;S size. Classification layers of each classifier receive feature maps having a feature dimension. An initial classification layer receives the feature maps as an output of a final feature extraction layer. Each classifier infers partition decisions for sub-blocks of size (&#x3b1;S)&#xd7;(&#x3b1;S) of the block, wherein &#x3b1; is a power of 2 and &#x3b1;=2, . . . , N/S, by applying, at some successive classification layers, a 1&#xd7;1 kernel to reduce respective feature dimensions; and outputting by a last layer of the classification layers an output corresponding to a N/(&#x3b1;S)&#xd7;N/(&#x3b1;S)&#xd7;1 output map.

Receptive-field-conforming convolution models for video coding

A convolutional neural network (CNN) for determining a partitioning of a block is disclosed. The block is of size N&#xd7;N and a smallest partition is of size S&#xd7;S. The CNN includes feature extraction layers; a concatenation layer that receives, from the feature extraction layers, first feature maps of the block, where each first feature map is of size S&#xd7;S; and classifiers. Each classifier includes classification layers, each classification layer receives second feature maps having a respective feature dimension. Each classifier is configured to infer partition decisions for sub-blocks of size (&#x3b1;S)&#xd7;(&#x3b1;S) of the block, wherein &#x3b1; is a power of 2 and &#x3b1;=2, . . . , N/S, by: applying, at some of successive classification layers of the classification layers, a kernel of size 1&#xd7;1 to reduce the respective feature dimension in half; and outputting by a last layer of the classification layers an output corresponding to a N/(&#x3b1;S)&#xd7;N/(&#x3b1;S)&#xd7;1 output map.

Motion estimation or compensation functionality of a hardware component is used to encode or decode key frames and other video frames. The hardware component includes a memory, which may, for example, be a local static random access memory or an external dynamic random access memory. Upon a block of a frame being encoded or decoded, data associated with that block is stored in the memory. That data can then be processed by motion estimation or motion compensation for use in encoding or decoding one or more later blocks within the same frame. The data may, for example, be stored in the memory after operations for reconstruction and loop filtering have been performed. The data stored in the memory may effectively be processed using traditional inter-prediction operations, such as to identify similar video objects within blocks of the same frame.

Growing community of inventors

Palo Alto, CA, United States of America

Aki Kuusela