Multi-modal embeddings of user interactions

Abstract

Aspects described herein may improve predictions of real-time user intent by generating multi-modal embeddings based on previous user interactions. The previous user interaction data may be received from multiple different modalities. The interaction data may be analyzed and labeled with user intent for each previous user interaction, then sequentially organized. The multi-modal embeddings may be generated by modal-specific encoders and a multi-modal embeddings generator. A trained machine learning model may use, as input, the multi-modal embeddings and personal data to determine real-time user intent at the beginning of an interaction. The multi-modal embeddings may act as a set of unique and specific output vectors that allow a computing device to recognize and correlate synonymous terms and interrelated subjects, usually a difficult task for computing devices, thereby improving the technological process of analyzing, in real-time, user intent.