International Business Machines Corporation

Couchbase, Inc.

Scott David Lashley

Kevin James Cherkauer

John Frederic Miller

Paul Sherwood Taylor

Jason Christopher Young

Roger C Raphael

Steven Robert Pearson

Raghupathi Keshava Murthy

Bingjie Miao

Clarence Madison Pruet, Iii

Effi Ofer

John Sae Liang

Srinath Duvuru

David James Oliver Rigby

Rohan Ashok Suri, Jr

Apaar Gupta

Sarath Lakshman

Xun Xue

Ronen Grosman

Gary Valentin

Roger L Q Zheng

Kevin Leo Beck

Dale Murray McInnis

Karl Ostner

Daniel Alan Wood

Paul Michael Brett

Jinming Xiao

Monish Gupta

Clarence Madison Iii Pruet

A system uses a hybrid key-value storage engine that uses log-structured merge tree and a segmented log-structured object store. The system performs garbage collection of stale document versions avoiding index lookup during log segment compaction. The system separates index and document data to minimize write amplification. The system maintains a delete list using a log-structured merge-tree to store stale document sequence numbers and corresponding sizes per log segment. For each log segment from the plurality of log segments, the system determines a measure of fragmentation of the log segment based on sizes of deleted documents of the log segment from the second log-structured merge-tree. If the fragmentation exceeds a threshold, the system initiates a compaction operation for the log segment.

Compaction of documents in a high density data storage system

High density data storage based on log structured storage techniques

According to embodiments of the present invention, machines, systems, computer-implemented methods and computer program products are provided for converting part or all of a document from an unstructured format into a structured format, and combining structured and unstructured data into an unstructured format. In some embodiments, documents are received in an unstructured format, and stored in a repository for querying (e.g., including updating and other operations (e.g., SQL SELECT, UPDATE, INSERT, DELETE, MERGE, etc.)). Responsive to detecting a predetermined number of the documents in an unstructured format stored in a repository for querying, an optimization criteria may be applied to identify one or more components of the stored documents for conversion, and the one or more components of the stored documents are converted from the unstructured format to a structured format.

Converting portions of documents between structured and unstructured data formats to improve computing efficiency and schema flexibility

Provided are techniques for building an undo log for in-memory blocks of data. Permission on a block of data in memory is set to prevent updates to that block of data using a memory protection function. In response to an update operation attempting to update the block of data in the memory, an interrupt with a location of the block of data is received, the block of data is copied to an undo log entry in an undo log, and the permission on the block of data in the memory is set to allow the update to that block of data to proceed using the memory protection function.

Building an undo log for in-memory blocks of data

An approach is provided in which a sample point system allocates sample point identifiers to a root node included an index tree that includes multiple leaf nodes. The sample point system distributes the sample point identifiers to the root node's child nodes, and recursively traverses through the index tree's hierarchical index levels and distributes the sample point identifiers from the child nodes to a subset of the index tree's leaf nodes. In turn, the sample point system collects sample data from the subset of the plurality of leaf nodes corresponding to the distributed sample point identifiers.

Gathering index statistics using sampling

A system for improved data sharing within a cluster of nodes having a database management system. The system defines a virtual replicated table as being useable in a hybrid of a shared-cache and shared-nothing architecture. The virtual replicated table is a physically single table sharable among a plurality of cluster nodes for data read operations and not sharable with other cluster nodes for data modification operations. Default owner node is assigned for each virtual replicated table to ensure the page validity and provide requested pages to the requesting node.

System using virtual replicated tables in a cluster database management system

A system for limiting the amount of time for a database server to perform a crash recovery process. A maximum recovery time for the database server to perform the crash recovery process is specified. An estimated recovery time for the crash recovery process that is less than the maximum recovery time is calculated, based on at least one of a deterministic analysis of cost accumulation during prior instances of the crash recovery process or an empirical analysis of cost accumulation during regular transaction processing in the database server. The crash recovery process is then conformed to the estimated recovery time.

System for deterministic database recovery time

Embodiments of the invention generally relate to data processing systems and more particularly to high availability data processing systems. A primary server may share a storage device with one or more clone systems. Each clone generally replicates the state of the primary server, but relies on the same disk-based storage as the primary server. Thus, the clone systems may provide a shadow of the primary server, ready to take over should the primary server fail. The clone systems may access a log file that includes entries reflecting the actions performed by the primary system. The primary server may flush entries from a log buffer maintained on the primary server to a log file stored on the shared disk-based storage. The primary server may also send a log sequence number to the clone systems, and the clone systems periodically transmit a log sequence number back to the primary server indicating how far through the log file a clone system has progressed.

Shared disk clones

A partial-blocking consistency point system identifies transaction updates with a consistency point ID associated with a consistency point sequence number, records consistency point data that identify a location of the partial-blocking consistency point, flushes to a non-volatile storage the transaction updates identified with the consistency point sequence number without blocking transaction activity, and hardens to the non-volatile storage the recorded partial-blocking consistency point so that data associated with the recorded partial-blocking consistency point can be recovered. The consistency point sequence number is incremented each time the partial-blocking consistency point is recorded to uniquely identify the partial-blocking consistency point and transaction updates associated with the partial-blocking consistency point. The transaction updates identified with the consistency point sequence number are processed to improve efficiency of the flushing of the transaction updates.

System and method for implementing a partial-blocking consistency point in a database

The present invention expounds upon the ANSI 'read committed' isolation level by allowing readers to read committed data without waiting for a concurrent writer to the data to finish. The method returns a last committed version of the data as it existed prior to changes made by the concurrent writer. Only two versions of any data record are required to be stored in the record data store, the last committed version and the current version. The last committed version may be generated from an undo log record. Locating the appropriate undo log record may be accomplished by storing a log sequence number in a lock data structure associated with the requested data record. A transaction flag may also stored in the lock data structure to facilitate generating the last committed version. The method may also utilize one or more locks to detect a concurrent writer to the requested data.

Computer program product for conducting a lock free read

A log-shipping data replication system employs a primary server coupled to a standby server. In operation, the primary server concurrently flushes log data at the primary server and transmits the same log data from the primary server to the standby server. The primary server further transmits at least one buffer flush boundary indication to the standby server indicating the flushing status of the log data at the primary server. The standby server processes the log data as a function of receiving the log data from the primary server, and/or receiving the at least one buffer flush boundary indication from the primary server.

Log shipping data replication with parallel log writing and log shipping at the primary site

A method and system are provided for providing a consistent reintegration of a failed primary instance as a new secondary instance with implementation of truncation of log records. Upon failure of a primary instance, a secondary instance may be reassigned as the new primary instance. Prior to reintegration, any portion of the database log of the failed primary that is after the log position of the new primary instance may be truncated, followed by a comparison of the log positions of both the new primary instance and the new secondary instance. In limited circumstances, the truncation of the log data generated by the failed primary instance after the point of failure is possible, and supports reintegration of the failed primary as a new secondary instance with limited overhead.

Growing community of inventors

Portland, OR, United States of America

Scott David Lashley