Intuit, Inc.

Tharathorn Joy Rimchala

Peter Lee Frick

Sricharan Kallur Palli Kumar

Guohan Gao

Terrence J Torres

Kamalika Das

Jiaxin Zhang

Shir Meir Lador

Yingxin Wang

Lalla Mouatadid

Dominic Miguel Rossi

Andrew Mattarella-Micke

Sameeksha Khillan

Hui Fang Lee

Yang Li

Deepankar Mohapatra

Conrad De Peuter

Preeti Duraipandian

Peter Caton Anthony

Priyadarshini Rajendran

Tak Yiu Daniel Li

Eric Vanoeveren

Amogha Sekhar

Karelia Del Carmen Pena Pena

Seth Harris Rubin

Jadiel De Armas

Harsha Ilapakurty

Subathra Sundararajan

Image pages are generated from a document. Assembling the image pages generates a collage. Two-dimensional text and bounding boxes are extracted from the image pages. A structure verbalizer spatially formats the two-dimensional text in one-dimension with spatial information to generate spatial-formatted text. The spatial-formatted text is concatenated to generate a text extraction. A multimodal embedding model is applied to the collage and the text extraction to generate a target artifact vector. The target artifact vector is compared against a set of preexisting artifact vectors to identify a corresponding artifact vector associated with a corresponding document having a corresponding metadata tag. A distance is determined between the corresponding artifact vector and the target artifact vector. Responsive to the distance being within a threshold distance additional steps are performed, including performing both applying the corresponding metadata tag to the document to generate a modified document and outputting the modified document.

Computationally efficient artifact tagging for document management

A method includes executing an encoder machine learning model on multiple token values contained in a document to create an encoder hidden state vector. A decoder machine learning model executing on the encoder hidden state vector generates raw text comprising an entity value and an entity label for each of multiple entities. The method further includes generating a structural representation of the entities directly from the raw text and outputting the structural representation of the entities of the document.

Entity extraction with encoder decoder machine learning model

A machine learning system executed by a processor may generate predictions for a variety of natural language processing (NLP) tasks. The machine learning system may include a single deployment implementing a parameter efficient transfer learning architecture. The machine learning system may use adapter layers to dynamically modify a base model to generate a plurality of fine-tuned models. Each fine-tuned model may generate predictions for a specific NLP task. By transferring knowledge from the base model to each fine-tuned model, the ML system achieves a significant reduction in the number of tunable parameters required to generate a fine-tuned NLP model and decreases the fine-tuned model artifact size. Additionally, the ML system reduces training times for fine-tuned NLP models, promotes transfer learning across NLP tasks with lower labeled data volumes, and enables easier and more computationally efficient deployments for multi-task NLP.

Method for serving parameter efficient NLP models through adaptive architectures

A computing system generates a plurality of training data sets for generating the NLP model. The computing system trains a teacher network to extract and classify tokens from a document. The training includes a pre-training stage where the teacher network is trained to classify generic data in the plurality of training data sets and a fine-tuning stage where the teacher network is trained to classify targeted data in the plurality of training data sets. The computing system trains a student network to extract and classify tokens from a document by distilling knowledge learned by the teacher network during the fine-tuning stage from the teacher network to the student network. The computing system outputs the NLP model based on the training. The computing system causes the NLP model to be deployed in a remote computing environment.

Methods and systems for generating mobile enabled extraction models

A visual-based classification model influenced by text features as a result of the outputs of a text-based classification model is disclosed. A system receives one or more documents to be classified based on one or more visual features and provides the one or more documents to a student classification model, which is a visual-based classification model. The system also classifies, by the student classification model, the one or more documents into one or more document types based on one or more visual features. The one or more visual features are generated by the student classification model that is trained based on important text identified by a teacher classification model for the one or more document types, with the teacher classification model being a text-based classification model. Generating training data and training the student classification model based on the training data are also described.

Text feature guided visual based document classifier

Aspects of the present disclosure provide techniques for image-based document search. Embodiments include receiving an image of a document and providing the image of the document as input to a machine learning model, where the machine learning model generates separate embeddings of a plurality of patches of the image of the document and the machine learning model generates an embedding of the image of the document based on the separate embeddings of the plurality of patches. Embodiments include determining a compact embedding of the image of the document based on applying a dimensionality reduction technique to the embedding of the image of the document generated by the machine learning model. Embodiments include performing a search for relevant documents based on the compact embedding of the image of the document. Embodiments include performing one or more actions based on one or more relevant documents identified through the search.

Image-based document search using machine learning

Systems and methods are disclosed for model based document image enhancement. Instead of requiring paired dirty and clean images for training a model to clean document images (which may cause privacy concerns), two models are trained on the unpaired images such that only the dirty images are accessed or only the clean images are accessed at one time. One model is a first implicit model to translate the dirty images from a source space to a latent space, and the other model is a second implicit model to translate the images from the latent space to clean images in a target space. The second implicit model is trained based on translating electronic document images in the target space to the latent space. In some implementations, the implicit models are diffusion models, such as denoising diffusion implicit models based on solving ordinary differential equations.

Model based document image enhancement

Certain aspects of the disclosure provide systems and methods for training an information extraction transformer model architecture directed to pre-training a first multimodal transformer model on an unlabeled dataset, training a second multimodal transformer model on a first labeled dataset to perform a key information extraction task processing the unlabeled dataset with the second multimodal transformer model to generate pseudo-labels for the unlabeled dataset, training the first multimodal transformer model based on a second labeled dataset comprising one or more labels, the pseudo-labels generated, or combinations thereof to generate a third multimodal transformer model, generating updated pseudo-labels based on label completion predictions from the third multimodal transformer model, and training the third multimodal transformer model using a noise-aware loss function and the updated pseudo-labels to generate an updated third multimodal transformer model.

Systems and methods for training an information extraction transformer model architecture

A system and method for extracting data from a piece of content using spatial information about the piece of content. The system and method may use a conditional random fields process or a bidirectional long short term memory and conditional random fields process to extract structured data using the spatial information.

System and method for spatial encoding and feature generators for enhancing information extraction

Systems and methods of generating synthetic training data for machine learning models. First, line items in source documents such as bills, invoices, and or receipts are identified and labeled. The identification and labeling generate labeled documents. Then, in the labeled documents, the line items are augmented by adding, deleting, and or swapping line items to generate synthetic training documents. An addition operation randomly selects one or more line items and adds the selected line item(s) to the same labeled document or another labeled document. A deletion operation randomly deletes one or more line items. A swapping operation randomly swaps line items in a single labeled document or across different labeled documents. These operations can generate synthetic labeled documents of any length, which form synthetic training data for training the machine learning models.

Compositional pipeline for generating synthetic training data for machine learning models to extract line items from OCR text

Certain aspects of the present disclosure provide techniques for training and using machine learning models to extract key-value sets from a document. An example method generally includes identifying regions of a document including key-value sets corresponding to inputs to a data processing application based on a first machine learning model and an electronic version of the document. One or more keys and one or more values are identified in the document based on a second machine learning model. One or more key-value sets are generated based on matching keys of the one or more keys and values of the one or more values in the region of the document. The one or more key-value sets are provided to a data processing application for processing.

Growing community of inventors

San Francisco, CA, United States of America

Tharathorn Joy Rimchala