Unified space-time interpolation of video information

Abstract

A technique is described herein for temporally and spatially interpolating input video information, to produce output video information having a higher frame rate and a higher resolution compared to that exhibited by the input video information. The technique generates feature information based on plural frames of the input video information. The technique then produces the output video information based on the feature information using an architecture having, in order, a multi-stage encoding operation, a query-generating operation, and a multi-stage decoding operation. Each encoding stage produces an instance of encoder attention information that expresses identified relations across the plural frames of the input video information. Each decoding stage operates on an instance of encoder attention information produced by a corresponding encoding stage. The transformer architecture is compact and is capable of interpolating the input video information in real time.