International Business Machines Corporation

Luis Garcés Erice

John G Rooney

Daniel Nikolaus Bauer

Peter Urbanetz

Florian Hermann Froese

Pascal Vetsch

Embodiments of the present invention provide methods, computer program products, and systems. For each dataset in the system, embodiments of the present invention can identify at least one stack of hierarchically-dependent system components on which that dataset is dependent, and generate a system graph having nodes interconnected by directed edges. Dependency of each dataset on a stack, and between successive components in each stack, is represented by a directed edge between nodes of the graph. Embodiments of the present invention can then calculate for each dataset, a stack-dependent compliance score as a function of the compliance scores for a set of nodes reached by traversing directed edges in the graph from the node corresponding to that dataset, and generate an alert if the stack-dependent compliance score for a dataset breaches a predetermined risk threshold for that dataset.

Compliance risk management for data in computing systems

A technique for retaining a context in which data resides independently of a data store from which the data originates is disclosed. In relation to a method aspect of the technique, a computer-implemented method provides data with related first metadata, both originating from a data store and extracts the data and the related first metadata independently from the data store. A universal unique identifier of a portion of the data to which portion specific first metadata exists is created as part of the related first metadata. The universal unique identifier of the portion of the data is integrated into the related first metadata, thereby creating modified first metadata as an independently manageable and linkable representation of the related first metadata.

Preserving metadata context in a hybrid cloud context

A method includes obtaining a first snapshot of a source system and deriving a set of key-value pairs reflecting the first snapshot. A mirror operation of the source system is performed to obtain CDC change operations. The method obtains a first CDC log as a first sequence of key-value pairs. A second snapshot of the source system is obtained and a set of key-value pairs is derived, which reflects the second snapshot. The first sequence of key-value pairs is compared with the set of key-value pairs to derive corrective CDC operations, captured as a set of key-value pairs. The corrective CDC operations represent corrections to be performed with respect to the first sequence of key-value pairs. A second CDC log is obtained as a second sequence of key-value pairs. The corrective CDC operations ensure that the second sequence of key-value pairs are coherent with the set of key-value pairs.

Tracking change data capture log history

An approach to finding data locations may be provided. A first synopses, which corresponds to first tabular data may be provided. An auxiliary data storage system may be scanned to identify second tabular data stored in the auxiliary data storage system. A second synopses may be obtained, in which the synopses correspond to the second columns of second tabular data. The synopsis may be computed for each second column of the second columns, according to a numeric representation of contents of cells of said each second column. The computed synopsis may include a vector of m descriptors. The two sets of one or more descriptors may be compared. A subset of the descriptors of the second synopses may be compared with corresponding descriptors of the first synopses, to identify potential matches between the second tabular data and the corpus of first tabular data.

Finding locations of tabular data across systems

A computer-implemented method for transferring changes made to data from a data source to a data destination may be provided. The method involves using a message broker with streaming channels for transferring data changes from a data source to a data consumer. To determine if data has changed, a portion of data labeled with a key is sent through a hash function. A hash value with the corresponding key that has already been sent to the data consumer is retrieved from a streaming channel and compared to the newly created hash value. If the newly created hash value is different the data is sent to data consumers by the message broker via one streaming channel, while the new hash value is stored in the other streaming channel.

Universal streaming change data capture

The present disclosure relates to a method for processing data of a database system, the database system being configured to connect to a centralized cache shared with other database systems. The method includes determining a unique identifier of the content of a table of the database system using a common identification method of the database systems. A database command may be generated for processing data of the table. The command references the identifier. It may be determined if the cache comprises results of a same command with the same identifier. The command may be processed at the database system or the results of the command may be retrieved from the cache depending on whether the cache comprises the results.

Processing data of a database system

A computer implemented method for enabling data coherency between a first data source and a second data source of different structure may be provided. The first data source has a first format and a related first log file. Similar conditions apply to the second data source. The method comprises also determining a first entry in the first log file at which the data have a semantic consistency and storing this as first read barrier. The same applies for an entry in the second data source. Additionally, the method comprises building a third data set derived from data of the first data source and data of the second data source and being coherent with them, using data from the first data source up to the first read barrier and data from the second data source up to the second read barrier.

Global coherency for distributed data sources

A computerized method for ingesting data from a relational database into a data lake is provided, wherein a user-defined function (UDF) is associated with a standard operation of extract, transform, load, or ETL, of an ETL pipeline. This UDF is triggered upon performing the standard operation and thereby allow a code associated with the UDF to be executed. Upon migrating data from one or more data sources into the relational database, the standard operation is executed, which triggers the UDF and, in turn, an execution of the code. As per the execution of this code, an entity running on the data lake is notified that a set of data migrated to the relational database is to be ingested according to given ingestion modalities specified by the code. Finally, the set of data can be ingested into the data lake according to the modalities. Related computer program products are also provided.

Growing community of inventors

Rueschlikon, Switzerland

Luis Garcés Erice