International Business Machines Corporation

Jonathan W Wierenga

Serge Bourbonnais

Xiao Li

Yat On Lau

Austin F M D'Costa

Hong Min

Christian Zentgraf

Si Bin Fan

Gong Su

Mai Zeng

Kan Zhang

Xiang Wei Zhou

Xin Wu

Somil Kulkarni

Marcel Kutsch

Tyrone Yu

Clarence M Pruet

Daniel B Morales

Transaction consistency query support is available for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Transaction consistency query support for replicated data from recovery log to external data stores

A method for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures is provided. The method includes loading data from an update-in-place data structure to a first set of hash buckets in a processing platform, loading data from append-only data structures to a second set of hash buckets in the processing platform, performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of has buckets, and generating a report based on the bucket-level comparison.

Verifying data consistency

Several replication subscriptions are defined for a table in a database management system. The table is divided into partitions. Each replication subscription replicates transactions to a range of partitions. Subscriptions are assignable to different consistency groups. Transaction consistency is preserved at the consistency group level. A persistent delay table is created for each of several apply functions. Each apply function processes replication subscriptions for one consistency group, to replicate the table to a target table in parallel. Transactions for a given range of a partition are executed in parallel. When an apply function upon a row of the target table results in an error, the row is stored in the delay table. Application of each row in the delay table is repeatedly retried, and if successful, the row is removed from the delay table.

Parallel replication of data table partition

A data replication method can begin with the detection of an inconsistency between records of a target table and corresponding records of a source table of a relational database management system (RDBMS) performing a parallel apply replication by an improved data replication manager. The target table can be a copy of the source table, both of which include multiple unique constraints and indexes. A timeframe that encompasses the records of the target table having the inconsistency can be determined. The timeframe can utilize a commit timestamp or a log sequence number. Consistency between the target table and the source table can be automatically restored for the determined timeframe through use of a reactive-apply process. Data suppression for updates is automatically restored once the copy is consistent. Transactions performed upon the target table by the reactive-apply process can be performed in parallel. Service at the source table and the target table can be uninterrupted.

Automatically restoring data replication consistency without service interruption during parallel apply

Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using the consistent change records.

Recovery log analytics with a big data management platform

A system, method and computer program product for enabling light weight table comparison with high-accuracy (high confidence) of tables where one is a copy of the other, which copy may be maintained synchronized by replication. The method performs database comparison using a sample-based, statistics-based, or materialized query tables-based approaches. The method first identifies a block comprising a sub-set of rows of data of a source database table and a corresponding block from a target database table, and obtains a statistical value associated with each block. Then the statistical values for the corresponding source and target block are compared and a consistency evaluation of source and target database is determined based on comparing results. Further methods enable a determination of the data as being persistent or not in manner that accounts for real-time data modifications to underlying source and target database tables while identified blocks are being compared.

Lightweight table comparison

Techniques are disclosed to determine differences between a source table and a target table in a database environment, as being persistent or transient. A first set of differences between the source table and the target table is determined at a first point in time. A second set of differences between the source table and the target table is determined at a second point in time subsequent to the first point in time. At least one of a set of persistent differences and a set of transient differences is determined. The set of persistent differences includes a set intersection of the first and second sets of differences, the set intersection being filtered based on matching non-key values of the differences. The set of transient differences includes a relative complement of the second set of differences in the first set of differences.

Difference determination in a database environment

According to one embodiment of the present invention, a system uses parallel transaction messages for database replication. The system receives transaction messages from a source system via a plurality of parallel send queues in a receive queue. Each transaction message includes a message identifier indicating a commit order for that transaction. The system reads transaction messages in the receive queue in order in which they were committed against a source database based on the message identifier, and applies changes described by the transaction messages to a target database. Two or more transaction messages in the receive queue are applied to the target database in parallel. The system deletes transaction messages from the receive queue asynchronously to applying changes described by those transaction messages to the target database. Embodiments of the present invention further include a method and computer program product for parallel transaction messages in substantially the same manners described above.

Parallel transaction messages for database replication

Techniques are disclosed for comparing database tables. In one embodiment, the database tables are partitioned. Queries are generated for retrieving each partition. For each generated query, a stored procedure is invoked, using the respective generated query as an input parameter to the stored procedure. The stored procedure is configured to generate a checksum based on the partition retrieved from executing the respective query. The application compares the generated checksums to determine if the partitions and/or tables are consistent.

Growing community of inventors

Wellington, New Zealand

Jonathan W Wierenga