Aspect ratio conversion with machine learning

Abstract

Techniques are disclosed for converting image frames, such as the image frames of a motion picture, from one aspect ratio to another while predicting the pan and scan framing decisions that a human operator would make. In one configuration, one or more functions for predicting pan and scan framing decisions are determined, at least in part, via machine learning using training data that includes historical pan and scan conversions. The training data may be prepared by extracting features indicating visual and/or audio elements associated with particular shots, among other things. Function(s) may be determined, using machine learning, that take such extracted features as input and output predicted pan and scan framing decisions. Thereafter, the image frames of a received video may be converted between aspect ratios on a shot-by-shot basis, by extracting the same features and using the function(s) to make pan and scan framing predictions.