Personalized generative video summarization

Abstract

A method includes generating a multimedia content result using a streaming multimodal video generation model, including creating prompt text based on one or more user preferences, generating image frames based on the prompt text, compiling the image frames into the multimedia content result, and displaying the multimedia content result. The method can also include acquiring video data and audio data associated with a multimedia content item, performing a first scene detection process on the video data, performing a second scene detection process on the audio data, determining combined scene detection results based on the first scene detection results and the second scene detection results, inputting at least a portion of the combined scene detection results into a genre-specific multimodal transformer (GMT) model, and generating the multimedia content result as a summarization for the multimedia content item using one or more outputs from the GMT model.