International Business Machines Corporation

Michael John Elvery Spicer

James Michael McArdle

Kevin Lynn Brown

Jing-Song Jang

Richard James Taylor

Wing Yun Mak

Method, system, and computer program product for transferring data from transactional data sources to partitioned databases are provided. One or more messages from a transactional data source are read. Each message includes one or more records to be inserted into a database comprising a plurality of partitions. One of the partitions of the database in which the one or more records of each message are to be inserted is then identified. A transaction is initiated to insert the one or more records of each message into the one partition and to delete the one or more messages from the transactional data source. The transaction is committed only when insertion of the one or more records of each message into the one partition and deletion of the one or more messages from the transactional data source succeed.

Transfer of data from transactional data sources to partitioned databases in restartable environments

Method, computer program product, and system for transferring data from positional data sources to partitioned databases are provided. A record is read from a positional data source. The record is to be written to one of a plurality of partitions of a database. A position of the record in the positional data source is obtained. A transaction is initiated to write the record to the one partition and to store the position of the record in the database. The transaction is committed after the record is successfully written to the one partition and the position of the record is successfully stored in the database.

Transfer of data from positional data sources to partitioned databases in restartable environments

Embodiments of the present invention provide methods and systems for high availability in data-stream applications using separate but synchronized nodes. Initially, a live node is designated to provide the data stream. Upon startup or restart, a standby node synchronizes its state with the live node. The initial synchronization is performed in two phases. In the first phase, the standby node connects to the same data stream (or a data stream with same content) and obtains the same symbols as the live node. In the second phase, the standby node obtains metadata from the live node for various symbols, such as symbols that are new to the standby node. Once synchronized, the live and the standby nodes then continue with separately processing the data stream.

Synchronizing in-memory caches while being updated by a high rate data stream

Method, computer program product, and system for transferring data to database partitions are provided. A transaction is initiated to insert one or more records into a partition of a database. At least one of the one or more records is then saved in binary form to a dump table in the database as a part of the transaction in response to the insertion of the at least one record into the partition having failed as a result of the at least one record not satisfying a predetermined condition. In response to the saving of the at least one record in binary form to the dump table having failed, the transaction is rolled back.

Handling transfer of bad data to database partitions in restartable environments

A system and associated method write data from an in-memory database to a disk database in an efficient manner and with a relatively short lag time. The integration of data from in-memory to disk is achieved by limiting the operations of the in-memory database to insert only. The system shortens lag time by reducing the number of transactions required to transfer data from in-memory database to disk memory. The system compiles into an RDBMS, knowledge about the structure of the in-memory database, and then uses the end of the transaction callbacks from the RDBMS to keep the in-memory database and the disk memory in synch. The RDBMS includes a daemon that runs periodically to find records in the in-memory database that have not yet been written to the RDBMS, and then writes the found records to the RDBMS as part of a single transaction. If the transaction completes successfully, the in-memory database is updated to reflect which records have been 'flushed' to the RDBMS. If the transaction fails, no action is taken. The present system synchronizes the in-memory database with the RDBMS.

System and method for efficiently writing data from an in-memory database to a disk database

A system and associate method handle out-of-order data supplied by a real-time feed, and ingests the real-time feed fast enough to keep up with the feed rate while storing the data in a database in a time-ordered or other sequential manner without discarding any data. The present system adds a second unordered list for out-of-order data received from the feed or from a replay feed. A data element received from the feed, which has a time stamp earlier than the last data element placed in the ordered list, is placed in the unordered list. If replay data is received, the replay data elements are placed in the unordered list without verifying the time stamp. The data is then flushed from these memory lists to a database. Both the ordered list and the unordered list are inputted into the database. The database handles the ordering and merging of these two lists on insertion. The ordered data is still available to the user for real-time query in memory while all data elements are available for analysis in the database.

System and method for handling out-of-order data supplied by a real-time feed

Methods, apparatus and data structures are described for representing, storing and manipulating date/time information in an 8-byte timestamp format. By limiting date/time information to a predetermined range and specifying a single date/time structure, an 8-byte timestamp is generated that is capable of conveying the same operational information as conventional 24-byte timestamps, yet which can be more efficiently generated, more efficiently stored, and more efficiently evaluated in boolean operations.

Growing community of inventors

Lafayette, CA, United States of America

Michael John Elvery Spicer