International Business Machines Corporation

Eastman-kodak Company

Marvin Mendelssohn

Hima Prasad Karanam

Mukesh Kumar Mohania

L Venkata Subramaniam

Tanveer Afzal Faruquie

Snigdha Chaturvedi

Sachindra Joshi

Albert Maier

Girish Venkatachaliah

Dan Dan Zheng

David M Ting

Tanveer A Faruquie

Angel Marie Smith

Robert M Smith

Heather Stimpson

Jan H Boer

Methods, systems, and computer program products for deriving a multi-pass matching algorithm for data de-duplication are provided herein. A method includes identifying multiple passes across multiple databases using a set of one or more blocking columns derived from a set of trained input data; identifying, in each of the multiple passes, one or more columns across the multiple databases that match one or more of the blocking columns; selecting a given pass from the multiple passes, wherein said given pass comprises a maximum number of matching columns within the multiple passes; determining, for the given pass, data that conform to the given pass comprising (i) a set of matching columns, (ii) one or more matching types and (iii) one or more weights; and determining one or more subsequent passes across the multiple databases iteratively by removing the data that conform to the given pass.

Deriving a multi-pass matching algorithm for data de-duplication

Computer program products and systems are provided for mining for sub-patterns within a text data set. The embodiments facilitate finding a set of N frequently occurring sub-patterns within the data set, extracting the N sub-patterns from the data set, and clustering the extracted sub-patterns into K groups, where each extracted sub-pattern is placed within the same group with other extracted sub-patterns based upon a distance value D that determines a degree of similarity between the sub-pattern and every other sub-pattern within the same group.

Automatically mining patterns for rule based data standardization systems

Methods, computer program products and systems are provided for mining for sub-patterns within a text data set. The embodiments facilitate finding a set of N frequently occurring sub-patterns within the data set, extracting the N sub-patterns from the data set, and clustering the extracted sub-patterns into K groups, where each extracted sub-pattern is placed within the same group with other extracted sub-patterns based upon a distance value D that determines a degree of similarity between the sub-pattern and every other sub-pattern within the same group.

A clustering-based approach to data standardization is provided. Certain embodiments take as input a plurality of addresses, identify one or more features of the addresses, cluster the addresses based on the one or more features, utilize the cluster(s) to provide a data-based context useful in identifying one or more synonyms for elements contained in the address(es), and standardize the address(es) to an acceptable format, with one or more synonyms and/or other elements being added to or taken away from the input address(es) as part of the standardization process.

Systems and methods for discovering synonymous elements using context over multiple similar addresses

Blocking column selection can include determining a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set, and determining a second parameter for each column set. The second parameter can indicate block size for the column set. For each column set, a measure of blockability that is dependent upon at least the first parameter and the second parameter can be calculated using a processor. The plurality of column sets can be ranked according to the measures of blockability.

Automatic selection of blocking column for de-duplication

A method of blocking column selection can include determining a first parameter for each column set of a plurality of column sets, wherein the first parameter indicates distribution of blocks in the column set, and determining a second parameter for each column set. The second parameter can indicate block size for the column set. For each column set, a measure of blockability that is dependent upon at least the first parameter and the second parameter can be calculated using a processor. The plurality of column sets can be ranked according to the measures of blockability.

An object-oriented document management and production system in which documents are represented as collections of logical components, or 'objects', that may be combined and physically mapped onto a page-by-page layout. Stored objects are organized, accessed and manipulated through a database management system. At a minimum, objects contain basic information-bearing constituents such as text, image, voice or graphics. Objects may also contain further data specifying appearance characteristics, relationships to other objects, and access restrictions.

Growing community of inventors

Melrose, MA, United States of America

Marvin Mendelssohn