Hybrid spatio-temporal neural models for video compression

Abstract

Systems and methods are provided for decoding visual content using a hybrid framework based on convolutional and neural radiance networks. A decoder receives bitstreams of model parameters, a sequence level representation, and a cross-resolution representation for reconstructing a sequence of frames. The model parameters comprise neural radiance network parameters. The decoder decodes the bitstreams of the model parameters, the sequence level representation, and the cross-resolution representation. The decoder generates, via a channel transformer, a combined representation based on the sequence level representation and the cross-resolution representation. The decoder adapts a neural network model based on the neural radiance network parameters. The decoder reconstructs the sequence of frames by determining, via the adapted neural network model based on the combined representation, pixel attribute information for each frame of the reconstructed sequence of frames. The decoder generates, for display at a client device, the reconstructed sequence of frames.