System and method for vision-assisted approach for graph structure extraction in various types of documents

Abstract

Various methods and processes, apparatuses or systems, and media for deterministically deriving underlying graph structure and associated text information in a document are disclosed. A processor implements a vision-based algorithm and a network-based algorithm that may extract and structure a diagram from an image obtained from the document. The processor deterministically derives underlying graph structure and associated text information in the document by applying the vision-based algorithm and the network-based algorithm, thereby allowing encoding of graph content and reasoning into downstream applications including LLM inputs, graphical question-answering, and information extraction tasks. The processor also implements OCR algorithm for text fields, and then isolates which piece of text belongs to which node by examining the spatial coordinates of the text against bounding box of the node and executes cross-page resolution.