Content-based video compression using reinforcement learning for video rate control

Abstract

A method for performing content-based video compression using reinforcement learning (RL) for video rate control for a downstream task is provided. The method includes processing frame information associated with a raw frame from a video using an RL agent to generate quantization parameter (QP) information that indicates one or more values associated with a compression level of the raw frame and encoding the raw frame into a bitstream based on the QP information. The method further includes reconstructing the raw frame using the bitstream to obtain a reconstructed frame and processing the raw frame as well as the reconstructed frame using a pre-trained downstream model to generate two outputs. The method then includes determining a downstream task reward based on the two outputs and training the RL agent based on the downstream task reward.