Generating a unifiled knowledge graph from multimodal data sources using collaborative multi-agent large language models

Abstract

Existing multi-agent system are employed collaboratively, where each agent specialized in extracting entities, relations, or events from documents which is a more generalized collaborative framework without specific customizations. The disclosed method presents a collaborative multi-agent knowledge graph Retrieval-Augmented Generation (RAG) framework using a plurality of collaborative multi-agent LLMs for extracting information from a plurality of multimodal documents and constructing the unified knowledge graph to facilitate accurate and efficient query-based retrieval. A query generator agent generates a plurality of queries that aim to uncover all possible information present in the plurality of multimodal documents. A domain model is generated by a domain model generator agent based on the plurality of queries. The domain model populator agent populates the domain model with data extracted from the plurality of multimodal documents. A unified knowledge graph is constructed from ta plurality of populated domain models and organizes the extracted information for accurate retrieval.