Deep model integration techniques for machine learning entity interpretation

Abstract

Various embodiments of the present disclosure provide machine learning training techniques for implementing a multi-modal interpretation process to generate holistic outputs for an event. The techniques may include generating, using first layers of a multi-modal machine learning model, text-based intermediate representations for an entity based on textual input data. The techniques include generating, using second layers of the multi-modal machine learning model, image-based intermediate representations for the entity based on the text-based intermediate representations and input images for the entity. The techniques include generating, using one or more third layers of the multi-modal machine learning model, an entity representation summary based on the image-based intermediate representations and an image narrative summary for the input images. The techniques include initiating the performance of a prediction-based action based on the entity representation summary.