Systems and methods for multi-modal conversation summarization on a conversation platform

Abstract

Embodiments described herein provide a multi-modal search-and-summarize tool for message platforms. Specifically, the multi-modal search-and-summarize tool may monitor conversational content of different formats, e.g., text, image, video, etc., and use multi-modal summarization models to generate a summary of the conversation channel. The summarization may be conducted via a search-and-summarize process in response to a specific user query, e.g., a user may enter 'what did John and Josh say about the presentation tomorrow?' The multi-modal summarization model would first search for relevant conversation messages between user John and user Josh, identify communication files of different format (e.g., text messages, emojis, multimedia attachments, etc.), and then input the communication files to respective text or image encoders to generate a summary of the communication content.