Adobe, Inc.

University System of Maryland

Vlad Ion Morariu

Rajiv Jain

Varun Manjunatha

Handong Zhao

Tong Sun

Christopher Alan Tensmeyer

Franck Dernoncourt

Curtis Michael Wigington

Jiuxiang Gu

Nikolaos Barmpalios

Ashutosh Mehra

Nedim Lipka

Kai Li

Joseph Barrow

Sasha Spala

Ani Nenkova

Nicholas Miller

Puneet Mathur

Nishant Sankaran

Scott David Cohen

Brian Lynn Price

Sumit Shekhar

Jason Wen Yong Kuen

Tong Yu

Zhicheng Liu

Quan Hung Tran

Anandhavelu N

Aparna Garimella

Vinay Aggarwal

Ruiyi Zhang

Bhanu Prakash Reddy Guda

Jayant Vaibhav Srivastava

Jason Kuen

Lars Niklas Emanuel Elmqvist

Verena Kaynig-Fittkau

Zhendong Chu

Vitali Petsiuk

Michael Brodie

Peizhao Li

Vicente Ignacio Ordonez Roman

Yuexi Chen

Kajal Gupta

Tushar Tiwari

Pramuditha Perera

Methods and systems disclosed herein relate generally to systems and methods for generating visual relationship graphs that identify relationships between objects depicted in an image. A vision-language application uses transformer encoders to generate a graph structure, in which the graph structure represents a dependency between a first region and a second region of an image. The dependency indicates that a contextual representation of the first region was derived, at least in part, by processing the second region. The contextual representation identifies a predicted identity of an image object depicted in the first region. The predicted identity is determined at least in part by identifying a relationship between the first region and other data objects associated with various modalities.

Self-supervised visual-relationship probing

This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that train a named entity recognition (NER) model with noisy training data through a self-cleaning discriminator model. For example, the disclosed systems utilize a self-cleaning guided denoising framework to improve NER learning on noisy training data via a guidance training set. In one or more implementations, the disclosed systems utilize, within the denoising framework, an auxiliary discriminator model to correct noise in the noisy training data while training an NER model through the noisy training data. For example, while training the NER model to predict labels from the noisy training data, the disclosed systems utilize a discriminator model to detect noisy NER labels and reweight the noisy NER labels provided for training in the NER model.

Generating an improved named entity recognition model using noisy data with a self-cleaning discriminator model

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate a digital document hierarchy comprising layers of parent-child element relationships from the visual elements. For example, for a layer of the layers, the disclosed systems determine, from the visual elements, candidate parent visual elements and child visual elements. In addition, for the layer of the layers, the disclosed systems generate, from the feature embeddings utilizing a neural network, element classifications for the candidate parent visual elements and parent-child element link probabilities for the candidate parent visual elements and the child visual elements. Moreover, for the layer, the disclosed systems select parent visual elements from the candidate parent visual elements based on the parent-child element link probabilities. Further, the disclosed systems utilize the digital document hierarchy to generate an interactive digital document from the digital document image.

Extracting document hierarchy using a multimodal, layer-wise link prediction neural network

The technology described includes methods for pretraining a document encoder model based on multimodal self cross-attention. One method includes receiving image data that encodes a set of pretraining documents. A set of sentences is extracted from the image data. A bounding box for each sentence is generated. For each sentence, a set of predicted features is generated by using an encoder machine-learning model. The encoder model performs cross-attention between a set of masked-textual features for the sentence and a set of masked-visual features for the sentence. The set of masked-textual features is based on a masking function and the sentence. The set of masked-visual features is based on the masking function and the corresponding bounding box. A document-encoder model is pretrained based on the set of predicted features for each sentence and pretraining tasks. The pretraining tasks includes masked sentence modeling, visual contrastive learning, or visual-language alignment.

Unified pretraining framework for document understanding

Certain embodiments involve using a machine-learning tool to generate metadata identifying segments and topics for text within a document. For instance, in some embodiments, a text processing system obtains input text and applies a segmentation-and-labeling model to the input text. The segmentation-and-labeling model is trained to generate a predicted segment for the input text using a segmentation network. The segmentation-and-labeling model is also trained to generate a topic for the predicted segment using a pooling network of the model to the predicted segment. The output of the model is usable for generating metadata identifying the predicted segment and the associated topic.

Machine-learning tool for generating segmentation and topic metadata for documents

Systems and methods for natural language processing are described. One or more embodiments of the present disclosure identify a claim from a document, wherein the claim corresponds to a topic, create a graph comprising a plurality of nodes having a plurality of node types and a plurality of edges having a plurality of edge types, wherein one of the nodes represents the claim, and wherein each of the edges represents a relationship between a corresponding pair of the nodes, encode the claim based on the graph using a graph convolutional network (GCN) to obtain an encoded claim, classify the claim by decoding the encoded claim to obtain a stance label that indicates a stance of the claim towards the topic, and transmit information indicating a viewpoint of the document towards the topic based on the stance label.

Syntopical reading for collection understanding

Systems and methods for document editing are provided. One aspect of the systems and methods includes obtaining a document and a natural language edit request. Another aspect of the systems and methods includes generating a structured edit command using a machine learning model based on the document and the natural language edit request. Yet another aspect of the systems and methods includes generating a modified document based on the document and the structured edit command, where the modified document includes a revision of the document that incorporates the natural language edit request.

Language-guided document editing

Adapting a machine learning model to process data that differs from training data used to configure the model for a specified objective is described. A domain adaptation system trains the model to process new domain data that differs from a training data domain by using the model to generate a feature representation for the new domain data, which describes different content types included in the new domain data. The domain adaptation system then generates a probability distribution for each discrete region of the new domain data, which describes a likelihood of the region including different content described by the feature representation. The probability distribution is compared to ground truth information for the new domain data to determine a loss function, which is used to refine model parameters. After determining that model outputs achieve a threshold similarity to the ground truth information, the model is output as a domain-agnostic model.

Domain adaptation for machine learning models

Systems and techniques for generating responsive documents are described. Digital content is organized into a structure that defines how content is presented when a document is displayed by a computing device. To generate the responsive document, relationships are defined among different digital content objects, such as groups of content objects to be presented together and content objects that are to be presented as alternatives of one another. Responsive patterns are assigned to grouped content objects, where each responsive pattern defines different layout configurations for displaying grouped content objects based on computing device display characteristics. In some implementations, multiple responsive patterns are assigned to a single content group and individual responsive patterns are associated with activation ranges for display characteristics that activate the responsive pattern. For groups of digital content objects that are assigned multiple responsive patterns, responsive patterns are prioritized to create a hierarchy dictating display of the responsive document.

Responsive document authoring

This disclosure describes one or more embodiments of systems, non-transitory computer-readable media, and methods that can learn or identify a learned-initialization-latent vector for an initialization digital image and reconstruct a target digital image using an image-generating-neural network based on a modified version of the learned-initialization-latent vector. For example, the disclosed systems learn a learned-initialization-latent vector from an initialization image utilizing a high number (e.g., thousands) of learning iterations on an image-generating-neural network (e.g., a GAN). Then, the disclosed systems can modify the learned-initialization-latent vector (of the initialization image) to generate modified or reconstructed versions of target images using the image-generating-neural network. For instance, the disclosed systems utilize the learned-initialization-latent vector as a starting point to learn a learned-latent vector for a target image that an image-generating-neural network converts into a high-fidelity reconstruction of the target image (with a reduced number of learning iterations).

Initializing a learned latent vector for neural-network projections of diverse images

One example method involves operations for a processing device that include receiving, by a machine learning model trained to generate a search result, a search query for a text input. The machine learning model is trained by receiving pre-training data that includes multiple documents. Pre-training the machine learning model by generating, using an encoder, feature embeddings for each of the documents included in the pre-training data. The feature embeddings are generated by applying a masking function to visual and textual features in the documents. Training the machine learning model also includes generating, using the feature embeddings, output features for the documents by concatenating the feature embeddings and applying a non-linear mapping to the feature embeddings. Training the machine learning model further includes applying a linear classifier to the output features. Additionally, operations include generating, for display, a search result using the machine learning model based on the input.

Self-supervised document representation learning

Embodiments provide systems, methods, and computer storage media for extracting semantic labels for field widgets of form fields in unfilled forms. In some embodiments, a processing device accesses a representation of a fillable widget of a form field of an unfilled form. The processing device generates an encoded input representing text and layout of a sequence of tokens in a neighborhood of the fillable widget. The processing device uses a machine learning model to extract a semantic label representing a field type of the fillable widget in view of the encoded input. The processing device causes execution of an action using the semantic label.

Automatic semantic labeling of form fields with limited annotations

The present disclosure relates to systems, non-transitory computer-readable media, and methods for classifying an input image utilizing a classification model conditioned by a generative model and/or self-supervision. For example, the disclosed systems can utilize a generative model to generate a reconstructed image from an input image to be classified. In turn, the disclosed systems can combine the reconstructed image with the input image itself. Using the combination of the input image and the reconstructed image, the disclosed systems utilize a classification model to determine a classification for the input image. Furthermore, the disclosed systems can employ self-supervised learning to cause the classification model to learn discriminative features for better classifying images of both known classes and open-set categories.

Classifying images utilizing generative-discriminative feature representations

A domain alignment technique for cross-domain object detection tasks is introduced. During a preliminary pretraining phase, an object detection model is pretrained to detect objects in images associated with a source domain using a source dataset of images associated with the source domain. After completing the pretraining phase, a domain adaptation phase is performed using the source dataset and a target dataset to adapt the pretrained object detection model to detect objects in images associated with the target domain. The domain adaptation phase may involve the use of various domain alignment modules that, for example, perform multi-scale pixel/path alignment based on input feature maps or perform instance-level alignment based on input region proposals.

Domain alignment for object detection domain adaptation tasks

Growing community of inventors

Potomac, MD, United States of America

Vlad Ion Morariu