International Business Machines Corporation

Rajesh M Desai

Roger C Raphael

Justo Luis Perez

Magesh Jayapandian

Varun Bhagwan

Paul Sherwood Taylor

Aidon Paul Jennery

Iun Veng Leong

Alon Shalev Housfater

Mu Qiao

Philip E Parker

Gabriel Valencia

Nazrul Islam

Ramani Ranjan Routray

Lei Zhang

William Russell Belknap

Ang Yi

Michael Busch

Ronald L Rathgeber

Piyoosh Jalan

Hai Cheng Wang

Daniel Gruhl

Jun Hong Zhao

Arjun Natarajan

Angineh Aghakiant

Yang Zhong Li

Sudhakar Basireddy

Robert A Foyle

Ashish Kundu

Jeffrey Alan Kusnitz

Scott Schumacher

Hani Talal Jamjoom

Fumihiko Terui

Ety Khaitzin

Ye Qing Chen

Sonali Surange

Jojo Joseph

Michael Kyle McCandless

Brian Joseph Owings

Joshua F Payne

Aniya Aggarwal

Mayank Vipin Shah

Tom William Jacopi

Xue Lan Zhang

Olena Woolf

Dong Rui Li

Xue Xu

Xue Mei Deng

Arron La

Lijing E Lin

Praveen Vyas

Hani Talal Jamjoon

Thomas Hampp

Satwik Hebbar

Uttama Shakya

Ramakanta Samal

Shalu Agrawal

Ansel Blume

Ayush Utkarsh

Xiaoying Gao

Shalin Avlani

A computer-implemented method for text block segmentation includes determining a first text block segmentation pattern utilized to generate a segmented text block based, at least in part, on a comparison of semantic information associated with the segmented text block and a plurality of predefined types of text block segmentation patterns indicated by a graph; calculating a first degree of confidence in a size of the segmented text block based, at least in part, on comparing semantic entities associated with the segmented text block with semantic entities indicated by leaf nodes stemming from a first non-leaf node included in the graph and representative of the first type of text block segmentation pattern; and determining that the size of the segmented text block is non-optimal based on the calculated degree of confidence in the size of the segmented text block being below a predetermined threshold.

Text block segmentation

A computer-implemented process for generating cluster templates used for creating extraction models includes the following operations. A plurality of training files associated with a selected class are received. An automated visual analysis is performed on each of the plurality of training files. An automated contextual analysis is performed on each of the plurality of training files. A first clustering of the plurality of training files into a first plurality of clusters using results from the automated visual analysis is performed. A second clustering of one of the plurality of clusters into a second plurality of clusters is performed using results from the automated contextual analysis. Cluster templates for the first and second plurality of clusters are generated.

Multi-layer approach to improving generation of field extraction models

Disclosed embodiments provide techniques for monitoring and evaluating the effectiveness of key value pairs (KVPs) used in a document processing system. In embodiments, KVPs are obtained from multiple extractors of a document processing system. A score is computed for the KVPs by computing an effectiveness metric for each KVP from the multiple KVPs. In response to the computed score being below a predetermined threshold, a model retraining process is performed to generate a new set of KVP extractors, and provide the new set of KVPs to the document processing system.

Dynamic optimization of key value pair extractors for document data extraction

In an approach, a processor identifies a plurality of text separators in a borderless table, a text separator of the plurality of text separators defining a non-text region between two consecutive text lines in the borderless table. A processor classifies the plurality of text separators into a number of target clusters comprised in a target group based on property information related to the plurality of text separators, the number of target clusters corresponding to a number of separator types. A processor provides indication information to indicate respective separator types of the plurality of text separators based on a result of the classifying.

Line item detection in borderless tabular structured data

A mechanism is provided for implementing an optical character recognition (OCR) error correction mechanism for correcting OCR errors. Responsive to receiving a document in which OCR has been performed, the mechanism assesses the document to identify a set of OCR errors generated by an OCR engine that performed the OCR using a set of visual embeddings. Responsive to identifying the set of OCR errors, the mechanism analyzes each character of a plurality of sentences within the document to generate a high-dimensional embedding for the characters of the plurality of sentences within the document. The mechanism then linguistically corrects each OCR error in the set of OCR error. The mechanism utilizes ground truth information and the set of visual embeddings to verify that character stream is linguistically correct. Responsive to verifying that the character stream is linguistically correct, the mechanism outputs an OCR error corrected document to a user.

Multi-modal learning based intelligent enhancement of post optical character recognition error correction

Discovering second-order documents and latent custodians in an e-discovery system is provided. A list of first-order documents and document custodians within a base state of the e-discovery system are identified based on a plurality of terms corresponding to a meet and confer practice for a legal matter instance. The plurality of terms is masked within the first-order documents. The first-order documents having the plurality of terms masked are divided into groups. A list of second-order documents is generated from a group of documents. A list of second-order document custodians is generated based on corresponding custodian relationships to second-order documents. Finally, each second-order document custodian in the list of second-order document custodians that has a corresponding rank exceeding a defined rank threshold level is identified as an official document custodian in the e-discovery system.

Discovering latent custodians and documents in an E-discovery system

Dynamic data dissemination is provided. A resolved data subject identifier corresponding to a data subject is selected from a set of resolved data subject identifiers existing in rows of a data asset. In response to determining that the resolved data subject identifier does not correspond to a right to forget list, it is determined that the resolved data subject identifier corresponds to a data subject request list. The rows are transformed to anonymize existing pseudo and personal identifiers in cells of the rows that are tied to columns associated with data classes for which specific consent dimensions have been indicated as revoked by the data subject.

Dynamic data dissemination under declarative data subject constraints

Preserving distributions of data values of a data asset in a data anonymization operation is provided. Anonymizing data values is performed by transforming sensitive data in a set of columns over rows of the data asset while preserving distribution of the data values in the set of transformed columns to a defined degree using a set of autoencoders and loss function. The autoencoders are base trained from preexisting data in a data assets catalog and actively trained during data dissemination. Parametric coefficients of the loss function are configured and the threshold is generated using policies from an enforcement decision for the data asset and data consumer. The loss function value of a selected row is compared to the threshold. Transformed data values of the selected row are transcribed to an output row when the loss function value is greater than the threshold and disseminated to the data consumer.

Learning to transform sensitive data with variable distribution preservation

Processing a database query for sets of data includes assigning a unique identifier from an integer space to each entity within data and creating one or more sets of entities each pertaining to a corresponding entity within the data. A representation is then generated on disk for each set of entities, wherein each representation encompasses and is suited for a range of the unique identifiers of entities within a corresponding set and indicates a presence of an entity within that corresponding set. Finally, a query is processed based on the representation for each set of entities to retrieve data satisfying the query, wherein the representation provides a constant time for association and dissociation operations that are append-only operations with deferred merge and automatic filtering of deleted and duplicate entities at query time.

Managing large scale association sets using optimized bit map representations

A method, apparatus, system, and computer program product evaluate an information asset with a corpus of policies in conjunction with the context of access including a specific user. A large corresponding set of rules in the policy corpus are identified by computer system. A continuous process of rule evaluation occurs against information asset metadata wherein a series of processing including set of common subexpressions between the predicates of all active rules, pre-evaluation, compaction and storage are identified by the computer system in the policy and rule corpus. Metadata for the information asset is applied by the computer system to the set of common subexpressions to form partially evaluated rules for the policy. The partially evaluated rules henceforth compacted are stored by the computer system in association with the information asset. Subsequently the partially evaluated rules are a compressed form of the rule corpus for the policies in the system and hence reduce computer resources and significantly improve the response tome used to evaluate the enforcement decision for the said information asset with the policy in conjunction with the context of access. The process can be repeated each time at least one of the metadata or the policy changes to provide continuous processing of rules to maintain compacted up-to-date partially evaluated rules to enforce the policy.

Real-time policy rule evaluation with multistage processing

Disclosed is a computer-implemented method to identify and anonymize personal information, the method comprising analyzing a first corpus with a personal information sniffer, wherein the first corpus includes unstructured text, wherein the personal information sniffer is configured to detect a set of types of personal information, and wherein the personal information sniffer produces a first set of results. The method comprises analyzing the first corpus with a set of annotators, wherein each annotator is configured to identify all instances of a type of personal information in the corpus, and wherein the set of annotators produces a second set of results. The method comprises comparing the first set of results and the second set of results, determining, the first set of results does not match the second set of results, and updating, based on the determining, the personal information sniffer.

Detecting and obfuscating sensitive data in unstructured text

A method, system and computer program product for detecting sensitive personal information in a storage device. A block delta list containing a list of changed blocks in the storage device is processed. After identifying the changed blocks from the block delta list, a search is performed on those identified changed blocks for sensitive personal information using a character scanning technique. After identifying a changed block deemed to contain sensitive personal information, the changed block is translated from the block level to the file level using a hierarchical reverse mapping technique. By only analyzing the changed blocks to determine if they contain sensitive personal information, a lesser quantity of blocks needs to be processed in order to detect sensitive personal information in the storage device in near real-time. In this manner, sensitive personal information is detected in the storage device using fewer computing resources in a shorter amount of time.

Detection of sensitive personal information in a storage device

Embodiments generally relate to providing litigation management for multiple remote content systems using asynchronous bi-directional replication pipelines. In some embodiments, a method includes retrieving, at one or more inbound replicators of one or more respective bi-directional pipelines, metadata associated with documents stored in one or more content repositories. The method further includes resolving, at a governance control hub, conflicts associated with legal holds on one or more of the documents based on the metadata. The method further includes sending conflict resolution results from one or more outbound applicators of the bi-directional pipelines to the content repositories, where the content repositories enforce legal holds on the documents.

Providing near real-time and effective litigation management for multiple remote content systems using asynchronous bi-directional replication pipelines

Serving data assets based on security policies is provided. A request to access an asset received from a user having a particular context is evaluated based on a set of asset access enforcement policies. An asset access policy enforcement decision is generated based on evaluating the request. It is determined whether the asset access policy enforcement decision is to transform particular data of the asset prior to allowing access. In response to determining that the asset access policy enforcement decision is to transform the particular data of the asset prior to allowing access, a transformation specification that includes an ordered subset of unit transformations for transforming the particular data of the asset is generated based on the particular context of the user and the set of asset access enforcement policies. A transformed asset is generated by applying the transformation specification to the asset transforming the particular data of the asset.

Serving data assets based on security policies by applying space-time optimized inline data transformations

A method, apparatus, system, and computer program product for evaluating enforcement decisions on an asset using a policy. Rules in the policy are applied by a computer system to the asset taking into account a context for a request to access the asset in response receiving to the request to access the asset, and wherein the rules in the policy determine whether access to the asset is allowed. A determination is made by the computer system as to whether a conflict is present in an initial decision made using the rules in the policy. A set of conflict resolution processes are applied by the computer system when the conflict is present such that a final decision is made on the request to access the asset.

Policy rule enforcement decision evaluation with conflict resolution

Growing community of inventors

San Jose, CA, United States of America

Rajesh M Desai