Wells Fargo Bank, N.a.

Tarun Joshi

Vijayan Narayana Nair

Rahul Singh

Agus Sudjianto

Harsh Singhal

Wayne B Shoumaker

Mina Naghshnejad

Angelina Yang

Yufei Yu

Karan Jindal

Matthew A Campbell

Hanyu Yang

Archit Parnami

Omkar Patil

A computing system includes a memory; and processing circuitry in communication with the memory. The processing circuitry is configured to: receive a paraphrase comprising a paraphrase text sample corresponding to an original text sample; and calculate a paraphrase metric value corresponding to the paraphrase, wherein the paraphrase metric value is calculated based on an adequacy score, a novelty score, and a fluency score of the paraphrase, the adequacy score indicating an extent to which the paraphrase text sample preserves a meaning of the original text sample, the novelty score indicating a level of difference between words and characters of the paraphrase text sample and words and characters of the original text sample, and the fluency score indicating an extent to which the paraphrase text sample is devoid of repetition, spelling, and grammatical mistakes.

Metric for assessing a quality of one or more paraphrases

An example electronic computing device can include: a processor; and a system memory, the system memory including instructions which, when executed by the processor, cause the electronic computing device to: receive a model for natural language processing of data, the model including a plurality of self-attention heads; prune the model by removing one or more of the plurality of self-attention heads of the model to create a pruned model; and evaluate a classification accuracy of the pruned model to maintain a performance level.

Compression of models for natural language processing

Disclosed in some examples are methods, systems, and machine readable mediums which provide summaries of topics determined within a corpus of documents. These summaries may be used by customer service associates, analysts, or other users to quickly determine both topics discussed and contexts of those topics over a large corpus of text. For example, a corpus of documents may be related to customer complaints and the topics may be summarized to produce summaries such as 'credit report update due to stolen identity.' These summarizations may be used to efficiently spot trends and issues.

Adversarial input generation for natural language processing machine learning models

Systems, apparatuses, methods, and computer program products are disclosed for determining robustness information for an NLP model. Modification rules, such as replacement rules and/or insertion rules, are used to generate instances of modified test data based on instances of test data that comprise words and have a syntax and a semantic meaning. The instances of test data and modified test data are provided to the NLP model and the output of the NLP model is analyzed to determine output changing instances of modified test data, which are instances of modified test data yielded output from the NLP model that is different and/or not similar to the output yielded from the NLP model for the corresponding instance of test data. Robustness information for the NLP model is determined based at least in part on the output changing instances of modified test data.

Systems and methods for natural language processing (NLP) model robustness determination

Systems, apparatuses, methods, and computer program products are disclosed for screening data instances based on a target text of a target corpus. A screening device analyzes a target corpus to generate at least two term dictionaries for the target corpus. The screening apparatus, based on a frequency of a term in the target corpus, determines a term weight for the term; for each data instance, determines term scores for the data instance and the target text based on the term weights; filters the data instances based on the term scores, to generate a short list of data instances; determines term similarity scores between each data instance of the short list and target text based on the term weights; and provides a data instance determined to likely correspond to the target text and an indication of the corresponding term similarity score(s). A term is a word or an n-gram.

Systems and methods for screening data instances based on a target text of a target corpus

Systems, apparatuses, methods, and computer program products are disclosed for at least semi-automated generation of model documentation. In an example embodiment, a template corresponding to a model is selected. User input populating portions of a build notebook is received. The build notebook comprises a model identifier identifying the model and executable code. The build notebook is executed using an execution engine to cause an intermediary file encoded in a type-setting system to be generated. Executing the build notebook causes cross-references of elements of the model documentation to be mapped to appropriate locations within the intermediary file. The elements comprise graphical representations of results of model tests executed on the model and/or extracted or user-defined artifacts required for the model documentation. The intermediary file is executed using a type-setting system compiler to assemble user-provided text and the elements, in accordance with the template, to generate the model documentation.

Model documentation generation system

Systems, apparatuses, methods, and computer program products are disclosed for determining robustness information for an NLP model. Modification rules, such as replacement rules and/or insertion rules, are used to generate instances of modified test data based on instances of test data that comprise words and have a syntax and a semantic meaning. The instances of test data and modified test data are provided to the NLP model and the output of the NLP model is analyzed to determine output changing instances of modified test data, which are instances of modified test data yielded output from the NLP model that is different and/or not similar to the output yielded from the NLP model for the corresponding instance of test data. Robustness information for the NLP model is determined based at least in part on the output changing instances of modified test data. White and/or black box attacks may be performed.

Growing community of inventors

San Francisco, CA, United States of America

Tarun Joshi