Nokia Technologies Oy

Nokia Corporation

Provenance Asset Group LLC

Francesco Cricri

Emre Baris Aksu

Miska Matias Hannuksela

Arto Juhani Lehtiniemi

Sujeet Shyamsundar Mate

Antti Johannes Eronen

Igor Danilo Diego Curcio

Jani Lainema

Hamed Rezazadegan Tavakoli

Jussi Artturi Leppänen

Honglei Zhang

Caglar Aytekin

Jussi Leppanen

Mikko Johannes Honkala

Xingyang Ni

Jukka Saarinen

Alireza Zare

Nam Le

Miika Sakari Tupala

Kostadin Dabov

Yat Hong Lam

Miikka Tapani Vilermo

Lasse Juhani Laaksonen

Juha Henrik Arrasvuori

Mikko Tapio Tammi

Mikko-Ville Laitinen

Sailesh Kumar Sathish

Antti Hallapuro

Iraj Saniee

Vinod Kumar Malamal Vadakital

Matti Sakari Hamalainen

Ville-Veikko Mattila

Radu Ciprian Bilcu

Joachim Wabnig

Jussi Leppänen

Ramin Ghaznavi Youvalari

Nannan Zou

Goutham Rangu

Mikko Roininen

Jukka Pentti Paivio Saarinen

Miika Tupala

Apparatuses, methods, and computer programs for compressing a neural network are disclosed. An apparatus includes at least one processor; and at least one non-transitory memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to: receive information from a second device, where the information comprises at least one parameter configured to be used for compression of a neural network, where the at least one parameter is in regard to at least one first aspect or task of the neural network; and compress the neural network, where the neural network is compressed based, at least partially, upon the at least one parameter received from the second device. The apparatus may also receive a compressed neural network from the second device, and further compress the compressed neural network based on the information.

High-level syntax for priority signaling in neural network compression

The embodiments relate to method for encoding and decoding, wherein the method for encoding comprises receiving an input block of a video frame for encoding; applying at least a learning-based model () for said input block as a processing step for encoding the block; combining () an output of a learning-based model with one or more data sources () by a combination process; encoding block to a bitstream (); using a result of the combination process as additional input for the learning-based model for encoding a subsequent block; and encoding to a bitstream combination information () used in the combination process, said combination information comprising at least one or more combination parameters. The embodiments also relate to technical equipment for implementing the methods.

Method, an apparatus and a computer program product for video encoding and video decoding

In example embodiments, an apparatus, a method, and a computer program product are provided. The apparatus comprises at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform determine a processing order of building blocks to encode or decode a media item; and determine a number of processing steps required to encode or decode the media item; wherein the processing order of building blocks and the number of processing steps are determined based on a content of the media item by using a guided probability model based on a neural network.

Guided probability model for compressed representation of neural networks

An apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: estimate an importance of parameters of a neural network based on a graph diffusion process over at least one layer of the neural network; determine the parameters of the neural network that are suitable for pruning or sparsification; remove neurons of the neural network to prune or sparsify the neural network; and provide at least one syntax element for signaling the pruned or sparsified neural network over a communication channel, wherein the at least one syntax element comprises at least one neural network representation syntax element.

Graph diffusion for structured pruning of neural networks

An apparatus including circuitry configured for: in a first-person perspective mediated reality state, rendering mediated reality content as content distributed across a first area of a user's field of view, wherein a point of view of a user determines a point of view within a three-dimensional virtual space and determines at least part of the content distributed across the first area of a user's field of view as a virtual scene; responding to at least one user gesture to enter a spatially consolidated state; and in the spatially consolidated state, rendering the mediated reality content as content distributed across a second area of a user's field of view, wherein the second area is smaller than the first area and the point of view of the user does not determine the content distributed across the second area of a user's field of view.

Mediated reality

Example embodiments provide a system for training a data coding pipeline including a feature extractor neural network, an encoder neural network, and a decoder neural network configured to reconstruct input data based on encoded features. A plurality of losses corresponding to different tasks may be determined for the coding pipeline. Tasks may be performed based on an output of the coding pipeline. A weight update may be determined for at least a subset of the coding pipeline based on the plurality of losses. The weight update may be configured to reduce a number of iterations for fine-tuning the coding pipeline for one of the tasks. This enables faster adaptation of the coding pipeline for one of the tasks after deployment of the coding pipeline. Apparatuses, methods, and computer programs are disclosed. Apparatuses, methods, and computer programs are disclosed.

Training a data coding system comprising a feature extractor neural network

The embodiments relate to a method comprising compressing input data (I) by means of at least a neural network (E,); determining a compression rate for data compression; miming the neural network (E,) with the input data (I) to produce an output data (c); removing a number of elements from the output data (c) according to the compression rate to result in a reduced form of the output data (me); and providing the reduced form of the output data (me) and the compression rate to a decoder (D,). The embodiments also relate to a method comprising receiving input data (me) for decompression; decompressing the input data (me) by means of at least a neural network (D,); determining a decompression rate for decompressing the input data (me); miming the neural network (D,) with input data (me) to produce a decompressed output data (ĩ); padding a number of elements to the compressed input data (me) according to the decompression rate to produce an output data (ĩ); and providing the output data (ĩ).

The embodiments relate to a method for encoding and a decoding, and apparatuses for the same. The method for encoding comprises receiving a block of a video frame for encoding (); making a decision on whether or not a learning-based model is to be applied as a processing step for encoding the block (); applying the learning-based model for said input block according to the decision, where the learning-based model has been selectively fine-tuned according to information relating to activation of the learning-based model of previously-decoded blocks (); encoding a signal corresponding to the decision on usage of the learning-based model into a bitstream (); and encoding the block into a bitstream with an information whether the block is to be used for finetuning ().

A method comprising: obtaining a configuration of at least one neural network comprising a plurality of intra-prediction mode agnostic layers and one or more intra-prediction mode specific layers, the one or more intra-prediction mode specific layers corresponding to different intra-prediction modes; obtaining at least one input video frame comprising a plurality of blocks; determining to encode one or more blocks using intra prediction; determining an intra-prediction mode for each of said one or more blocks; grouping blocks having same intra-prediction mode into groups, each group being assigned with a computation path among the plurality of intra-prediction mode agnostic and the one or more intra-prediction mode specific layers; training the plurality of intra-prediction mode agnostic and/or the one or more intra-prediction mode specific layers of the neural networks based on a training loss between an output of the neural networks relating to a group of blocks and ground-truth blocks, wherein the ground-truth blocks are either blocks of the input video frame or reconstructed blocks; and encoding a block using a computation path assigned to an intra-prediction mode for the block.

Apparatus, a method and a computer program for video coding and decoding

Media content is received for streaming to a user device. A neural network is trained based on a first portion of the media content. Weights of the neural network are updated to overfit the first portion of the media content to provide a first overfitted neural network. The neural network or the first overfitted neural network is trained based on a second portion of the media content. Weights of the neural network or the first overfitted neural network are updated to overfit the second portion of the media content to provide a second overfitted neural network. The first portion and the second portion of the media content are sent with associations to the first overfitted neural network and the second overfitted to the user equipment.

Content-specific neural network distribution

A method includes maintaining a set of parameters or weights derived through online learning for a neural net; transmitting an update of the parameters or weights to a decoder; deriving a first prediction block based on an output of the neural net using the parameters or weights; deriving a first encoded prediction error block through encoding a difference of the first prediction block and a first input block; encoding the first encoded prediction error block into a bitstream; deriving a reconstructed prediction error block based on the first encoded prediction error block; deriving a second prediction block based on an output of the neural net using the parameters or weights and the reconstructed prediction error block; deriving a second encoded prediction error block through encoding a difference of the second prediction block and a second input block; and encoding the second encoded prediction error block into a bitstream.

Data may be encoded to minimize distortion after decoding, but the quality required for presentation of the decoded data to a machine and the quality required for presentation to a human may be different. To accommodate different quality requirements, video data may be encoded to produce a first set of encoded data and a second set of encoded data, where the first set may be decoded for use by one of a machine consumer or a human consumer, and a combination of the first set and the second set may be decoded for use by the other of a machine consumer or a human consumer. The first and second set may be produced with a neural encoder and a neural decoder, and/or may be produced with the use of prediction and transform neural network modules. A human-targeted structure and a machine-targeted structure may produce the sets of encoded data.

Cascaded prediction-transform approach for mixed machine-human targeted video coding

An apparatus includes circuitry configured to: partition an input tensor into one or more block tensors; partition at least one of the block tensors into one or more continuation bands, the one or more continuation bands being associated with a caching counter having a value; store the one or more continuation bands in a cache managed using a cache manager; retrieve, prior to a convolution or pooling operation on a current block tensor, the one or more continuation bands of a previous block tensor from the cache that are adjacent to a current block tensor; concatenate the retrieved continuation bands with the current block tensor; apply the convolution or pooling operation on the current block tensor after the concatenation; decrease the respective caching counter value of the retrieved continuation bands; and clear the continuation bands from the cache when its respective caching counter reaches a value of zero.

Caching and clearing mechanism for deep convolutional neural networks

A method, apparatus, and computer program product are provided for training a neural network or providing a pre-trained neural network with the weight-updates being compressible using at least a weight-update compression loss function and/or task loss function. The weight-update compression loss function can comprise a weight-update vector defined as a latest weight vector minus an initial weight vector before training. A pre-trained neural network can be compressed by pruning one or more small-valued weights. The training of the neural network can consider the compressibility of the neural network, for instance, using a compression loss function, such as a task loss and/or a weight-update compression loss. The compressed neural network can be applied within a decoding loop of an encoder side or in a post-processing stage, as well as at a decoder side.

Compressing weight updates for decoder-side neural networks

A method comprising: obtaining a block of a picture or a picture in an encoder; determining if the block/picture is used for on-line learning; if affirmative, encoding the block/picture; reconstructing a coarse version of the block/picture or the respective prediction error block/picture; enhancing the coarse version using a neural net; fine-tuning the neural net with a training signal based on the coarse version; determining if the block/picture is enhanced using the neural net; and if affirmative, encoding the block/picture with enhancing using the neural net.

Growing community of inventors

Tampere, Finland

Francesco Cricri