System and method for layout-aware generative pretraining for visually rich document understanding

Abstract

Various methods and processes, apparatuses/systems, and media for performing spatial-aware reading for visual documents are disclosed. A processor implements a language model; modifies existing parameters and architecture of the language model to incorporate new parameters in its architecture by integrating a disentangled spatial attention process to the language model; pretrains the modified model by performing an autoregressive block infilling process on a plurality of document pages thereby training the new parameters and further adjusting the existing parameters; instruction-tunes the pretrained model on data derived from a plurality of visually rich document understanding datasets to teach the pretrained model to follow document-oriented instructions or answer questions about documents by leveraging their content and their spatial layout and outputting a trained model; and performs spatial-aware reading for visual documents by utilizing the trained model.