Semantic multimodal search of engineering documents

Abstract

In example embodiments, a semantic multimodal search function is provided in an engineering application for searching engineering documents. In an indexing phase, the application indexes a library of engineering documents to build an embedding database. For each of the engineering documents, a region detector extracts regions that each correspond to a different mode of technical or engineering data, and a set of ML models is used to generate embeddings that represent the semantic significance of technical or engineering data in each of the regions. In a query phase, the application receives search input and uses the region detector to extract regions that each correspond to a different mode of input. The application uses the ML models to generate embeddings that represent the semantic significance of input in each of the regions, and then performs a vector search between these embeddings and those in the embedding database.