Adobe, Inc.

Ashutosh Mehra

Vlad Ion Morariu

Rajiv Jain

Mohit Gupta

Uttam Dwivedi

Md Nadeem Akhtar

Jayant Vaibhav Srivastava

Pranav Kumar

Franck Dernoncourt

Shawn Alan Gaither

Nedim Lipka

Sumit Shekhar

Curtis Michael Wigington

Varun Manjunatha

Jiuxiang Gu

Quan Hung Tran

Aparna Garimella

Anandhavelu N

Vinay Aggarwal

Ani Nenkova

Bhanu Prakash Reddy Guda

Puneet Mathur

Kajal Gupta

Vitali Petsiuk

Tushar Tiwari

Vicente Ignacio Ordonez Roman

Verena Kaynig-Fittkau

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate a digital document hierarchy comprising layers of parent-child element relationships from the visual elements. For example, for a layer of the layers, the disclosed systems determine, from the visual elements, candidate parent visual elements and child visual elements. In addition, for the layer of the layers, the disclosed systems generate, from the feature embeddings utilizing a neural network, element classifications for the candidate parent visual elements and parent-child element link probabilities for the candidate parent visual elements and the child visual elements. Moreover, for the layer, the disclosed systems select parent visual elements from the candidate parent visual elements based on the parent-child element link probabilities. Further, the disclosed systems utilize the digital document hierarchy to generate an interactive digital document from the digital document image.

Extracting document hierarchy using a multimodal, layer-wise link prediction neural network

Embodiments provide systems, methods, and computer storage media for extracting semantic labels for field widgets of form fields in unfilled forms. In some embodiments, a processing device accesses a representation of a fillable widget of a form field of an unfilled form. The processing device generates an encoded input representing text and layout of a sequence of tokens in a neighborhood of the fillable widget. The processing device uses a machine learning model to extract a semantic label representing a field type of the fillable widget in view of the encoded input. The processing device causes execution of an action using the semantic label.

Automatic semantic labeling of form fields with limited annotations

A method comprises determining instance bounds associated with each of one or more structural elements in a document using a machine learning model. The method further comprises determining an error in the instance bounds associated with a particular one of the one or more structural elements. The method further comprises correcting the error in the instance bounds associated with the particular structural element using document content associated with the particular structural element, thereby generating corrected instance bounds associated with the particular structural element. The method further comprises generating a structural map of the document based on the corrected instance bounds.

Document structure identification using post-processing error correction

Various disclosed embodiments can resolve output inaccuracies produced by many machine learning models. Embodiments use content order as input to machine learning model systems so that they can process documents according to the position or rank of instances in a document or image. In this way, the model is less likely to misclassify or incorrectly detect instances or the ordering between predicted instances. The content order in various embodiments can be used as an additional signal to classify or make predictions.

Machine learning prediction and document rendering improvement based on content order

Techniques are disclosed for identifying document structural elements and correcting errors in the classification and/or location of the identified structural elements. An example method includes determining location and classification for a structural element on a page of the document using a machine learning (ML) model; determining one or more errors in the location and/or classification for the structural element; and correcting each instance of the one or more errors using other content in the document (e.g., content spatially adjacent to the corresponding structural element on the page of the document). The method may further include storing the document and the location and classification (as corrected), and/or generating a structural map of the page of the document based on the location and classification (as corrected). The use of the document content to correct errors greatly enhances the agreement between the identified structural elements and the original document.

Introduced here are computer programs and associated computer-implemented techniques for creating visualizations to explain the outputs produced by models designed for object detection. To accomplish this, a graphics editing platform can obtain a reference output that identifies a region of pixels in a digital image that allegedly contains an object. Then, the graphics editing platform can compute the similarity between the reference output and a series of outputs generated by a model upon being applied to masked versions of the digital image. A visualization component can be produced based on the similarity.

Explanatory visualizations for object detection

Techniques described herein implement heading identification and classification for a digital document in a digital medium environment. A document analysis system is leveraged to extract structural features from a digital document, identify heading candidates from among the structural features, validate the headings candidates, and classify validated headings into different headings types. The classified headings are then utilized to generate a sectioned version of the digital document ('sectioned document') that is divided into different sections based on the headings. Further, a document directory is generated that includes the headings and that enables navigation to different sections of the sectioned document.

Heading identification and classification for a digital document

Techniques described herein implement identifying artifacts in digital documents in a digital medium environment. A document analysis system is leveraged to extract page features from a digital document and to determine whether certain page features represent page artifacts such as headers and footers. Those page features determined to be page artifacts can be extracted from the digital document to generate a reflowed version of the digital document that preserves primary content. The primary content, for instance, is rearranged in the reflowed document to compensate for the extracted page artifacts.

Identifying artifacts in digital documents

Growing community of inventors

Noida, India

Ashutosh Mehra