International Business Machines Corporation

Pinar Tozun

Vijayshankar Raman

Ronald Jason Barber

Richard Sefton Sidle

Adam J Storm

Ronen Grosman

Christian Marcelo Garcia-Arellano

Rene Mueller

Guy Maring Lohman

Yuanyuan Tian

Mir Hamid Pirahesh

Chandrasekaran Mohan

Matthew Albert Huras

Chen Luo

Matthew J Spilchen

One embodiment provides for a method including performing, by a processing thread, a process that analyzes transactional operations by maintaining the transactional operations in transaction local side logs, and waiting until a successful transaction commit to append the transaction local side logs to a log stream. The processing thread processes the transactional operations on a key used to determine whether existing data is found for the key. The transactional operations are sped up through parallelism based on partitioning tables across nodes handling the transactional operations. A first process is performed by a first processor that processes updates for values of a key based on updating a first start time table index using unique keys and a start time field of a row for a first appearance of each unique key from the transactional operations.

Resolving versions in an append-only large-scale data store in distributed data management systems

According to one or more embodiments of the present invention, a computer-implemented method includes creating a hybrid index for a multi-zone data storage system in which a first zone has data that is organized for transaction processing, and a second zone that has data organized for analytic processing. Each index entry in the hybrid index includes an index key including an equality column, a begin-timestamp column, and a tuple-sequence-number column. The method further includes building an index run to be included in the hybrid index, the index run includes multiple index entries. An index run is created by selecting columns by scanning data blocks from the first zone, computing tuple-sequence-number values for the index entries to be included in the hybrid index, and sorting the index entries according to a predetermined order using the columns to be included in the index run.

Indexing for evolving large-scale datasets in multi-master hybrid transactional and analytical processing systems

A computer-implemented method for processing a transaction may include, upon receiving a transaction change request, operating a coordinator to generate a unique transaction identifier for the transaction request, send a data change request along with the unique transaction identifier to sources, and store the unique transaction identifier along with a respective first change count in a transaction status table. The method may also include, upon receiving a query, operating a given source to determine from the transaction status table, a second change count of sources associated with the unique transaction identifier, determine whether the second change count matches the first change count for each source, and when the first and second change counts match in each of the other sources, read data associated with the query from the given source and each source.

Count and transaction identifier based transaction processing

One embodiment provides for a method for grooming transactional operations in a multi-master distributed data management system including maintaining transactional operations in at least one transaction-local side-log. A table log scan is performed for a table including recording identification for each record in a log. Tuple sequence numbers are generated for the records and the records are written on groomed transaction blocks to the table.

Transactional operations in multi-master distributed data management systems

One embodiment provides for a method including processing transactional operations on a key used to determine whether existing data is found for that key. A first time index is updated using unique keys and a start time field of a first appearance of each key from the transactional operations. A deferred update of prior versions of the key is performed for non-recent data upon determining that recent data in the transactional operations is found for the key.

One embodiment provides a method for efficient representation, access and modification of variable length data objects including converting, by a processor, all object updates into appends. The processor divides each object update into bounded size tokens. The tokens are stored normalized as an anchor and a linked list of fragments, the anchor having a pointer to a linked list of fragments. A fragment comprises a contiguous list of tokens that correspond to the appends.

Efficient representation, access and modification of variable length objects

One embodiment provides a method comprising adjusting a runtime of a dataflow processing environment to operate on multiple batches of objects. The method further comprises pre-allocating one or more vectors of objects, and processing the multiple batches one at a time. The one or more vectors of objects are re-used during processing of each batch.

Efficient dataflow processing for objects

One embodiment provides a method for transparent partial object instantiation for object oriented applications including analyzing, by a processor, application code to determine a list of objects. The processor determines a list of accessed fields for each of the objects based on performing analysis for each function in the application code for accessed fields and other functions invoked.

Partial object instantiation for object oriented applications

A computer-implemented method for processing a transaction may include, upon receiving a transaction change request, operating a coordinator to generate a unique transaction identifier for the transaction request, send a data change request along with the unique transaction identifier to sources having respective source identifiers, and store the unique transaction identifier along with the respective source identifiers in a transaction status table. The computer-implemented method may further include, upon receiving a query, operating a given source to determine from the transaction status table, any other sources associated with the unique transaction identifier, determine whether the unique transaction identifier is in each of the other sources, and when the unique transaction identifier is in each of the other sources, read data associated with the query from the given source and each of the other sources.

Unique transaction identifier based transaction processing

One embodiment provides for a method for grooming transactional operations in a multi-master distributed data management system including maintaining transactional operations in at least one transaction-local side-log. A table log scan is performed for a table including recording identification for each record in a log. Each column in the table is loaded with column values based on the identifications. Records based on the column values are written to the groomed transaction blocks that are table based, and the tuple sequence numbers for each record on groomed blocks are generated. The table is shared/sharded across nodes handling transactional operations in the multi-master distributed data management system to increase processing handling of the transactional operations.

Growing community of inventors

San Jose, CA, United States of America

Pinar Tozun