International Business Machines Corporation

Mauricio Antonio Hernandez-Sherrington

Lucian Popa

Yunyao Li

Ching-Tien Ho

Lingling Yan

Mary Ann Roth

Rajasekar Krishnamurthy

Craig Salter

Laura Myers Haas

Renee J Miller

Jayatheerthan Krishnamurthy

Ramiya Venkatachalam

Min Li

Shivakumar Vaithyanathan

Oktie Hassanzadeh

Hima Prasad Karanam

Huaiyu Zhu

Georgia Koutrika

Douglas Ronald Burdick

Ronald Fagin

Daina Edvina Pupons Wickham

Marina Danilevsky Hailpern

Armageddon Rhabdizo Brown

Kun Qian

Suresh Thalamati

Bogdan Alexe

Ryan Wisnesky

Robert LaVerne Hobbs

Nikita Bhutani

Michael James Wehar

Li Qian

Kiranmayi Potu

Felix Gerhard Naumann

Mary Anne Roth

Data records are linked across a plurality of datasets. Each dataset contains at least one data record, and each data record is associated with an entity and includes one or more attributes of that entity and a value for each attribute. Values associated with attributes are compared across datasets, and matching attributes having values that satisfy a predetermined similarity threshold are identified. In addition, linkage points between pairs of datasets are identified. Each linkage point links one or more pairs of data records. Each data record in the pair of data records is contained in one of a given pair of datasets, and each pair of data records is associated with a common entity having matching attributes in the given pair of datasets. Data records associated with the common entities are linked across datasets using the identified linkage points.

Discovery of linkage points between data sources

Methods, systems, and computer program products for constraint tracking and inference generation are provided herein. A computer-implemented method includes parsing descriptions of one or more user-provided constraints pertaining to data within a target system, parsing truth value assignments to the user-provided constraints, and deriving a truth value for at least one of the user-provided constraints that does not correspond to a known truth value, wherein said deriving comprises performing a logical inference utilizing known truth values of one or more of the user-provided constraints. The computer-implemented method also includes storing, in a database, (i) the user-provided constraints, (ii) the known truth values, and (iii) the at least one derived truth value, and outputting the at least one derived truth value, one or more identified contradictions among the known truth values, and/or an indication that one or more unknown truth values corresponding to the user-provided constraints remain unknown.

Constraint tracking and inference generation

Methods, systems, and computer program products for entity structured representation and variant generation are provided herein. A computer-implemented method includes automatically parsing instances of a given entity type into semantic components by implementing a parser based at least in part on (i) the given entity type and (ii) items of information relevant to the given entity type; generating, based at least in part on (i) the semantic components and (ii) information pertaining to one or more valid component-specific variants, one or more variants of the semantic components; creating, based at least in part on the one or more variants of the one or more semantic components, one or more variants of at least one instance of an entity associated with the given entity type; and outputting, to at least one user, the one or more variants of the at least one instance of the entity.

Entity structured representation and variant generation

One embodiment provides a method including: receiving input indicating at least one concept from at least one document of an input document collection; generating a validated data model for each of a plurality of concepts, wherein each of the concepts is represented as a visual data structure comprising semantics; generating at least one intermediate model object by parsing the validated data model, wherein each of the intermediate model objects comprises a concept object and the at least one intermediate model object identifies concept dependencies; translating the at least one intermediate model object into executable source code by importing at least one pre-built extractor and translating at least one rule identified from the visual data structure; and generating an executable information extraction program from the executable source code, wherein the generating comprises generating at least one rule for the executable information extraction program based on the identified concept dependencies.

Translation of a visual representation into an executable information extraction program

Described herein are methods, systems and computer program products for entity resolution. Entity resolution, also known as entity matching or record linkage, seeks to identify equivalent data objects between or among datasets. An example method includes creating a deterministic model by defining an entity to be resolved, selecting two datasets for comparison, defining matching predicates for attributes of the datasets to select a set of candidate matches, and defining a precedence rule for the candidate matches to select a subset of the candidate matches. The method includes running the deterministic model on the two datasets. Running the deterministic model includes applying the matching predicates and the precedence rule to data in the datasets that correspond to the attributes. The method also includes applying a cardinality rule to results of the running, and outputting the matching candidates for which the cardinality rule is satisfied.

Entity resolution between datasets

One embodiment provides a method for generating an executable extraction program from a visual representation, the method including: utilizing at least one processor to execute computer code that performs the steps of: obtaining a validated model comprising a plurality of concepts, wherein each of the concepts comprises semantics associated with a visual representation of an object; generating at least one intermediate model object by parsing the validated model, wherein each of the intermediate model objects comprises a concept object; translating the at least one intermediate model object into executable source code; and generating an executable information extraction program from the executable source code. Other aspects are described and claimed.

Various embodiments of a method, system and article of manufacture to discover relationships among a first set of elements and a second set of elements are provided. At least one metric algorithm is identified based on a metric selection parameter. A raw result is determined based on the at least one metric algorithm, a first specified structural description of the first set of elements and a second specified structural description of the second set of elements. The raw result comprises a plurality of relationship measurements and the raw result is ordered. In some embodiments, a balanced result is produced based on the raw result and a matching strategy algorithm. In other embodiments, the matching strategy algorithm is identified based on a matching strategy selection parameter.

Tolerant and extensible discovery of relationships in data using structural information and data analysis

Embodiments of the present invention relate to a declarative framework for efficient incremental information integration. In one embodiment, a method of and computer program product for information integration is provided. An integration rule is received. A first data set is accessed. A first representation of the first data is generated set based on the plurality of integration rules. The first representation is flat and includes a plurality of records. At least one index is generated. The index encodes at least one hierarchical relationship among the plurality of records. A second representation is generated of the first representation based on the at least one index. The second representation comprising nested data.

Incremental information integration using a declarative framework

Embodiments of the present invention relate to a new method of entity integration using high-level scripting languages. In one embodiment, a method of and computer product for entity integration is provided. An entity declaration is read from a machine readable medium. The entity declaration describes an entity including at least one nested entity. An index declaration is read from a machine readable medium. The index declaration describes an index of nested entities. An entity population rule is read from a machine readable medium. The entity population rule describes a mapping from an input schema to an output schema. The output schema conforms to the entity declaration. A plurality of input records is read from a first data store. The input records conform to the input schema. The entity population rule applies to the plurality of records to create a plurality of output records complying with the output schema. An index of nested entities is populated. The index complies with the index declaration.

Entity integration using high-level scripting languages

A method, system and program product for specifying, in a schema mapping framework, a mapping between a source schema and a target schema. The source and target schemas are schemas included in respective groups of registered, heterogeneous schemas. The source and target schemas may be of different types. Serialized versions of the source and target schemas include source objects and target objects, respectively. A mapping model is serialized into mapping objects that include logical references representing the source objects and logical references representing the target objects. The logical references are resolved to the source objects and target objects, thereby storing pointers to the source objects and to the target objects. After resolving the logical references, the mapping model includes the logical references and the pointers to the source and target objects.

Schema mapping specification framework

Techniques are provided for viewing mappings between objects. A main view is displayed, wherein the main view shows one or more source objects, one or more target objects, and zero or more mappings between the one or more source objects and the one or more target objects. Input selecting a type of view to be displayed in the main view is received, wherein each type of view provides a different amount of detail regarding the mappings. In response to receiving the input, the selected type of view is created, and the created view is displayed. Additionally, techniques are provided for viewing objects. One or more objects along with mappings between the one or more objects are displayed. View filters are provided that may be applied to the one or more objects, wherein the view filters enable hiding at least one of mapped or unmapped objects. Moreover, techniques are provided for viewing nodes. A structure is displayed that includes one or more nodes. A hide option is provided that may be applied to the one or more displayed nodes, wherein selection of the hide option hides one or more selected nodes of the displayed nodes and descendants of the selected nodes.

Techniques for viewing mappings and nodes

A method and system for specifying, in a schema mapping framework, a mapping between a source schema and a target schema. The source and target schemas are schemas included in respective groups of registered, heterogeneous schemas. The source and target schemas may be of different types. Serialized versions of the source and target schemas include source objects and target objects, respectively. A mapping model is serialized into mapping objects that include logical references representing the source objects and logical references representing the target objects. The logical references are resolved to the source objects and target objects, thereby storing pointers to the source objects and to the target objects. After resolving the logical references, the mapping model includes the logical references and the pointers to the source and target objects.

A method and system for generating a query implementing a schema mapping. A mapping M is provided from a schema S to a schema T, where M relates S to T, and M includes a plurality of constraints. Schemas S and T each include one or more elements, and T includes at least one set type element. Mapping M is expressed in terms of at least one nested tuple-generating dependency. A query Q is generated where Q is capable of applying M to an input instance I to result in an output instance J, where I conforms to S, J conforms to T, and I and J satisfy the plurality of constraints. Instance J is in partitioned normal form (i.e., satisfies minimal union semantics) and includes no duplicate element instances.

Mapping-based query generation with duplicate elimination and minimal union

A computer program product is provided that uses data examples as a basis for understanding and refining declarative schema mappings. The system of the present invention identifies a set of intuitive operators for manipulating examples including establishing value correspondences, data linking, data trimming, data walking, and data chasing. These operators allow a user to follow and refine an example by walking through a data source. In addition, these operators can identify a large class of schema mappings and distinguish effectively between alternative schema mappings. With these operators, a user is able to quickly and intuitively build and refine complex data transformation queries that map one data source into another while continuously verifying that the mapping is accurate and appropriate.

Method for schema mapping and data transformation

A system and method for generating a source schema-to-target schema mapping includes grouping user-defined value correspondences into potential sets such that, for each potential set, at most one value correspondence per target attribute exists. The value correspondences represent functions and filters for deriving values of target attributes from values of source attributes. Candidate sets are selected from the potential sets and then grouped into ranked covers. Using the best cover, an SQL query representing a source schema-to-target schema mapping is generated.

Growing community of inventors

Gilroy, CA, United States of America

Mauricio Antonio Hernandez-Sherrington