Performing semantic segmentation of form images using deep learning

Abstract

The present disclosure relates to generating fillable digital forms corresponding to paper forms using a form conversion neural network to determine low-level and high-level semantic characteristics of the paper forms. For example, one or more embodiments applies a digitized paper form to an encoder that outputs feature maps to a reconstruction decoder, a low-level semantic decoder, and one or more high-level semantic decoders. The reconstruction decoder generates a reconstructed layout of the digitized paper form. The low-level and high-level semantic decoders determine low-level and high-level semantic characteristics of each pixel of the digitized paper form, which provide a probability of the element type to which the pixel belongs. The semantic decoders then classify each pixel and generate corresponding semantic segmentation maps based on those probabilities. The system then generates a fillable digital form using the reconstructed layout and the semantic segmentation maps.