Engagement prediction network based on video content

Abstract

Technological improvements are described for predicting user engagement with newly published, un-viewed, relatively short videos based on features extracted from the video content itself. A network includes an extraction module for extracting features from the video content and encoding the text-based features. The visual features are processed using learnable, per-frame and per-clip multi-layer perceptrons (MLPs) and using multi-modal cross-attention mechanisms. A fusion module merges a set of predicted features based on the results generated by the MLPs. A temporal aggregation module uses a multi-layer self-attention architecture to combine the fused features and obtain a set of temporal aggregated features. The network uses two branches (consisting of MLP layers) to jointly predict two new engagement metrics: a normalized action watch percentage (NAWP) and an engagement continuation rate (ECR). A recommendation system uses the predicted NAWP and the predicted ECR to generate a recommendation associated with the un-viewed video.