Sas Institute Inc.

Teresa S Jade

Xu Yang

Fan Wang

Xiao Dong Li

Qin Yang

Chunqi Zuo

Paul Jeffrey Kovach

Wei-shan Chiang

Aaron Douglas Arthur

Seng Lee

Julia Moreno

Ashley Mary Beck

Techniques described herein provide for generation of structured output for documents identified using automated near-duplicate detection. In one example, a system can receive a set of documents including at least one pair of similar documents determined to be similar to one another based on similarity scores generated using a predefined similarity scoring technique. The system can generate document groups by merging together pairs of documents that share at least one document. The system can, for each of the document groups, identify a representative document for the document group. The system can generate an output for display including a section for each document group, in which each section includes the representative document for the document group and, for each document in the document group, the similarity score relative to the representative document for the document group.

Structured output of duplicate or near-duplicate text documents identified using automated near-duplicate detection for text documents

Techniques described herein provide for automated near-duplicate detection for new text documents given text documents that were previously processed using automated near-duplicate detection for text documents. In one example, a system can receive new documents and documents that were previously processed using a predefined processing technique for automated near-duplicate detection. The system can process the new documents and cluster the new documents into multiple predefined clusters previously identified using the predefined processing technique. For each predefined cluster including at least one new document, the system can generate document groups by determining similarity scores using the predefined processing technique as applied to the documents in the predefined clusters. The system can identify a representative document for each document group and generate an output data structure including the document groups and the representative document for each group.

Expediting automated near-duplicate detection for new text documents

A computer-program product obtains a text document that includes text describing an action. The computer-program product extracts one or more action tokens from the text document. The computer-program product executes a plurality of linguistic pattern searches that search the text document for one or more likelihood tokens associated with the one or more action tokens. The computer-program product classifies the action to a likelihood category associated with a respective linguistic pattern search of the plurality of linguistic pattern searches. The computer-program product classifies the text document to a respective domain. The computer-program product computes a priority value of the action described in the text document. The computer-program product generates a priority summary artifact that visually prioritizes the text document over one or more other text documents when the priority value of the action satisfies a predefined maximum priority threshold value.

Systems, methods, and graphical user interfaces for predicting and analyzing action likelihood

Techniques described herein provide for automated detection of near-duplicate documents. In one example, a system can cluster documents into a set of clusters based on character frequencies associated with the documents. For a given cluster, the system can generate first similarity scores associated with every pair of documents in the cluster. The system can then select a filtered group of documents associated with first similarity scores that meet or exceed a first predefined similarity threshold. Next, the system can convert the filtered group of documents into matrix representations. The system can generate second similarity scores for every pair of matrix representations. The system can then identify documents, from among the filtered group of documents, associated with second similarity scores that meet or exceed a second predefined similarity threshold. The identified documents can be duplicate or near-duplicate text documents.

Automated near-duplicate detection for text documents

Text profiles can be leveraged to select and configure models according to some examples described herein. In one example, a system can analyze a reference textual dataset and a target textual dataset using text-mining techniques to generate a first text profile and a second text profile, respectively. The first text profile can contain first metrics characterizing the reference textual dataset and the second text profile can contain second metrics characterizing the target textual dataset. The system can determine a similarity value by comparing the first text profile to the second text profile. The system can also receive a user selection of a model that is to be applied to the target textual dataset. The system can then generate an insight relating to an anticipated accuracy of the model on the target textual dataset based on the similarity value. The system can output the insight to the user.

Leveraging text profiles to select and configure models for use with textual datasets

A human language analyzer receives, at the human language analyzer, text data representing information in a human language. The human language analyzer receives a computer command for identifying a text data component of the text data. The computer command comprises at least two requirements for the text data component. The human language analyzer, responsive to identifying that the first requirement and the second requirement are met, locates the text data component from one of two clauses. A clause analyzer receives a clause request to locate clauses within text data representing information in a human language. The clause analyzer receives, responsive to a dependency request, token information in a token data set. The clause analyzer determines a location for each clause of the sentence portion in a hierarchy of clauses. The clause analyzer generates and outputs a new data set based on the token data set and the hierarchy of clauses.

Human language analyzer for detecting clauses, clause types, and clause relationships

A human language analyzer receives, at the human language analyzer, text data representing information in a human language. The human language analyzer receives a computer command for identifying a text data component of the text data. The computer command comprises at least two requirements for the text data component. The human language analyzer, responsive to identifying that the first requirement and the second requirement is met, locates the text data component from one of two clauses. A clause analyzer receives a clause request to locate clauses within text data representing information in a human language. The clause analyzer receives, responsive to a dependency request, token information in a token data set. The clause analyzer determines a location for each clause of the sentence portion in a hierarchy of clauses. The clause analyzer generates and outputs a new data set based on the token data set and the hierarchy of clauses.

Growing community of inventors

Cary, NC, United States of America

Teresa S Jade