Hybrid spatio-temporal neural models for video compression

Abstract

Systems and methods are provided for encoding visual content using a hybrid framework based on convolutional and neural radiance networks. An encoder accesses video data having a sequence of frames. The encoder generates a first frame based on averaging pixel attributes of the sequence of frames. The encoder determines a sequence level representation based on the first frame. The encoder trains a neural network model based on the sequence of frames to determine a cross-resolution representation corresponding to the sequence of frames. Training the neural network model comprises generating a plurality of model parameters for reconstructing the sequence of frames based on the sequence level representation and the cross-resolution representation. The plurality of model parameters comprises neural radiance network parameters. The encoder transmits bitstreams of the plurality of model parameters, the sequence level representation, and the cross-resolution representation.