Method and system of using domain specific knowledge in retrieving multimodal assets

Abstract

A system for retrieving multimodal assets using domain-specific knowledge includes receiving a search query for searching for multimodal assets; encoding the search query into a first query representation via a first trained query representation machine-learning (ML) model and a second query representation via a second trained query representation ML model; comparing the first query representation to a plurality of multimodal representations to calculate a first similarity score, each of the plurality of multimodal representations being a representation of one of the plurality of candidate multimodal assets; comparing the second query representation to a plurality of domain-specific representations to calculate a second similarity score, the domain-specific representations being representations of domain-specific data associated with one or more of the plurality of the multimodal representations; calculating a third similarity score based on keyword matching between the domain-specific data and the one or more search terms in the search query; aggregating the first, second and third similarity scores to calculate a total similarity score for each of the plurality of candidate multimodal assets; ranking the plurality of candidate multimodal assets based on the total similarity scores to identify search results for the search query; and providing the identified candidate multimodal assets for display as the search results.