International Business Machines Corporation

Serge Bourbonnais

Xiao Li

Jonathan W Wierenga

David B Petersen

Paul Michael Cadarette

Michael G Fitzpatrick

Yat On Lau

Gregory Walter Vance

Hong Min

Austin F M D'Costa

Bruce Gilbert Lindsay

Somil Kulkarni

Christian Zentgraf

You-Chin Gene Fuh

Gong Su

Elizabeth Belva Hamel

Si Bin Fan

Stephen James Todd

James Z Teng

Mai Zeng

Miao Zheng

Pamela McLean

Tyrone Yu

Jing Cheng

Ke Wei Wei

Kan Zhang

Yi Jin

Xi Long Zheng

Hong Jing Ying

Xiang Wei Zhou

Austin D'costa

John Glenn Thompson

Charles Daniel Wolfson

Xin Wu

Nan Jiang

Tuong Chanh Truong

John Simon Tilling

Marcel Kutsch

Martin W Cocks

Nicolas G Adiba

David M Mooney

Roger L Zheng

Siqun Li

David A Clitherow

Siquan Li

Chengfei Liu

Jens Stankiewitz

Zhen Hua Dong

Roger Zheng

En Zhong Wang

Clarence M Pruet

David Mooney

Daniel B Morales

A method, system and computer program product is provided. The method includes identifying a large statement, writing rows that are changed by the large statement into a set of files outside a database management system recovery log, and writing log records for the set of files to the database management system recovery log, such that the log records provide ordering of row changes made by the large statement. The log records include a log sequence number corresponding to a first row changed and a format and encoding used for the row changes. The method further includes streaming, by the one or more processors of the computer system, the set of files out-of-band by a replication capture process to a target database management system.

Replicating large statements with low latency

Transaction consistency query support is available for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.

Transaction consistency query support for replicated data from recovery log to external data stores

A method, system and computer program product is provided. The method includes identifying a large statement, writing the rows that are changed by the large statement into a set of files outside a database management system recovery log, and writing log records for the set of files to the database management system recovery log, such that the log records provide ordering of row changes made by the large statement. The log records include a log sequence number corresponding to a first row changed and a format and encoding used for the row changes. The method further includes streaming, by the one or more processors of the computer system, the set of files out-of-band by a replication capture process to a target database management system.

A system for maintaining a two-site configuration for continuous availability over long distances may include a first computing site configured to execute a first instance associated with a priority workload, the first instance being designated as an active instance; a second computing site configured to execute a second instance of the priority workload, the second instance being designated as a standby instance; a software replication module configured to replicate a unit of work data associated with the priority workload from a first data object associated with the active instance to a second data object associated with the standby instance, and a hardware replication module configured to replicate an image from a first storage volume to a copy on a second storage volume, wherein the first storage volume is associated with the first computing site, and the second storage volume is associated with a third computing site.

Maintaining two-site configuration for workload availability between sites at unlimited distances for products and services

Systems for replication group partitioning include a workload profiling module configured to analyze historical workload data for a plurality of data elements to identify and categorize one or more transaction patterns; and a recommendation module configured to generate a recommended partitioning of the plurality of data elements into one or more replication groups, based on the one or more transaction patterns, that are optimized toward a partitioning goal.

Replication group partitioning

Methods and systems for replication group partitioning include analyzing historical workload data for a plurality of data elements to generate one or more transaction patterns and generating a recommended partitioning of the plurality of data elements into one or more replication groups, based on the one or more transaction patterns, that are optimized toward a partitioning goal.

A method includes reading, by a capture engine, log data. The log data can be synchronously hardware-replicated. The log data reflects that an update was started for a database of a first site. The capture engine corresponds to a capture engine of a second site. The method also includes identifying, by the capture engine, the update. The method also includes determining whether the update was committed to the database of the first site. The method also includes replicating, by an apply engine, the update to a database of the second site based on the determining. The update is replicated to the database of the second site based on a determination that the update was committed to the database of the first site. The committed update corresponds to a consistent replication of updates across two or more databases that reside in multiple database management systems.

Ensuring consistent replication of updates in databases

A method for verifying data consistency between update-in-place data structures and append-only data structures containing change histories associated with the update-in-place data structures is provided. The method includes loading data from an update-in-place data structure to a first set of hash buckets in a processing platform, loading data from append-only data structures to a second set of hash buckets in the processing platform, performing a bucket-level comparison between the data in the first set of hash buckets and the data in the second set of has buckets, and generating a report based on the bucket-level comparison.

Verifying data consistency

Several replication subscriptions are defined for a table in a database management system. The table is divided into partitions. Each replication subscription replicates transactions to a range of partitions. Subscriptions are assignable to different consistency groups. Transaction consistency is preserved at the consistency group level. A persistent delay table is created for each of several apply functions. Each apply function processes replication subscriptions for one consistency group, to replicate the table to a target table in parallel. Transactions for a given range of a partition are executed in parallel. When an apply function upon a row of the target table results in an error, the row is stored in the delay table. Application of each row in the delay table is repeatedly retried, and if successful, the row is removed from the delay table.

Parallel replication of data table partition

A system for providing continuous availability comprises a second computing site that is geographically remote from a first computing site. The second computing site comprises a storage unit configured to store units of work data and data synchronously replicated from the first computing site. The second computing site further comprises a group of workloads configured to process the units of work data asynchronously replicated from the first computing site. The second computing site further comprises a software replication module configured not to replicate any data while the first computing site is available.

Zero-data loss recovery for active-active sites configurations

Techniques for initializing a connection in a peer-to-peer replication environment. Responsive to a request to add a target node to a data replication group, embodiments receive a first message, at a target node, from an initiator node within the data replication group, indicating that the initiator node will begin replicating changes to the target node without waiting for any acknowledgements relating to the first message. The changes are applied to a first instance of a data entity on the initiator node. A second message is transmitted, from the target node, to at least one other member in the data replication group, announcing that the target node is joining the data replication group. Embodiments receive a plurality of replicated changes, at the target node, from the initiator node, and process each replicated change in the plurality of replicated changes against a second instance of the data entity at the target node.

Initialization protocol for a peer-to-peer replication environment

A data replication method can begin with the detection of an inconsistency between records of a target table and corresponding records of a source table of a relational database management system (RDBMS) performing a parallel apply replication by an improved data replication manager. The target table can be a copy of the source table, both of which include multiple unique constraints and indexes. A timeframe that encompasses the records of the target table having the inconsistency can be determined. The timeframe can utilize a commit timestamp or a log sequence number. Consistency between the target table and the source table can be automatically restored for the determined timeframe through use of a reactive-apply process. Data suppression for updates is automatically restored once the copy is consistent. Transactions performed upon the target table by the reactive-apply process can be performed in parallel. Service at the source table and the target table can be uninterrupted.

Automatically restoring data replication consistency without service interruption during parallel apply

Provided are techniques for replicating relational transactional log data to a big data platform. Change records contained in change data tables are fetched. A relational change history with transaction snapshot consistency is rebuilt to generate consistent change records by joining the change data tables and a unit of work table based on a commit sequence identifier. The consistent change records are stored on the big data platform, and queries are answered on the big data platform using the consistent change records.

Recovery log analytics with a big data management platform

Growing community of inventors

Palo Alto, CA, United States of America

Serge Bourbonnais