International Business Machines Corporation

Other

Mauricio A Hernandez

Lucian Popa

Oktie Hassanzadeh

Ching-Tien Ho

Salvatore J Stolfo

Rajasekar Krishnamurthy

Georgia Koutrika

Armageddon Rhabdizo Brown

Suresh Thalamati

Ryan Wisnesky

Data records are linked across a plurality of datasets. Each dataset contains at least one data record, and each data record is associated with an entity and includes one or more attributes of that entity and a value for each attribute. Values associated with attributes are compared across datasets, and matching attributes having values that satisfy a predetermined similarity threshold are identified. In addition, linkage points between pairs of datasets are identified. Each linkage point links one or more pairs of data records. Each data record in the pair of data records is contained in one of a given pair of datasets, and each pair of data records is associated with a common entity having matching attributes in the given pair of datasets. Data records associated with the common entities are linked across datasets using the identified linkage points.

Methods and systems for discovery of linkage points between data sources

Embodiments of the present invention relate to a new method of entity integration using high-level scripting languages. In one embodiment, a method of and computer product for entity integration is provided. An entity declaration is read from a machine readable medium. The entity declaration describes an entity including at least one nested entity. An index declaration is read from a machine readable medium. The index declaration describes an index of nested entities. An entity population rule is read from a machine readable medium. The entity population rule describes a mapping from an input schema to an output schema. The output schema conforms to the entity declaration. A plurality of input records is read from a first data store. The input records conform to the input schema. The entity population rule applies to the plurality of records to create a plurality of output records complying with the output schema. An index of nested entities is populated. The index complies with the index declaration.

Entity integration using high-level scripting languages

The semantic integration problem for merging multiple databases of very large size, the merge/purge problem, can be solved by multiple runs of the sorted neighborhood method or the clustering method with small windows followed by the computation of the transitive closure over the results of each run. The sorted neighborhood method works well under this scheme but is computationally expensive due to the sorting phase. An alternative method based on data clustering that reduces the complexity to linear time making multiple runs followed by transitive closure feasible and efficient. A method is provided for identifying duplicate records in a database, each record having at least one field and a plurality of keys, including the steps of sorting the records according to a criteria applied to a first key; comparing a number of consecutive sorted records to each other, wherein the number is less than a number of records in said database and identifying a first group of duplicate records; storing the identity of the first group; sorting the records according to a criteria applied to a second key; comparing a number of consecutive sorted records to each other, wherein the number is less than a number of records in said database and identifying a second group of duplicate records; storing the identity of the second group; and subjecting the union of the first and second groups to transitive closure.

Method of merging large databases in parallel

Growing community of inventors

San Jose, CA, United States of America

Mauricio A Hernandez