Deep video understanding with large language models

Abstract

Disclosed herein are system, apparatus, article of manufacture, method and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for deep video understanding with large language models. An example embodiment operates by determining a relationship between respective first and second visual elements for each of a plurality of frames of a content item based on respective element types and respective locations for the respective first and second visual elements. For each of the plurality of frames, a respective visual prompt is generated describing the relationship between the respective first and second visual elements. Based on an audio-to-text conversion of audio content associated with the frame or classification of aural elements of the audio content, a respective audio prompt describing the audio content associated with each frame is generated. A description of the content item is output by a large language model (LLM) based on the visual prompts and audio prompts for the plurality of frames input to the LLM.