Salesforce Com, Inc.

Salesforce, Inc.

Riverbed Technology, Inc.

Oracle International Corporation

Bea Systems, Inc.

James E Mace

Patrick James Helland

Thomas Fanghaenel

Nathaniel Wyatt

Punit B Shah

Kaushal Mittal

Subho Sanjay Chatterjee

Nitin Parab

Michael Richards

Akshay Manchale Sridhar

Steven McCanne

Mark Wilding

Jameison Bear Martin

Boris Krasnoiarov

Atish Agrawal

Suhas Dantkale

Jun Chen

Shao-Yuan Ho

David B Anderson

Piyush Kumar Jain Mahendra Kumar

Matthew Woicik

Techniques are disclosed relating to backing up skip list data structures to facilitate a subsequent recovery. In various embodiments, a computing system creates a checkpoint of a skip list including a plurality of key-value records that include pointers to others of the plurality of key-value records. Creating the checkpoint includes scanning the skip list to identify ones of the plurality of key-value records that are relevant to the checkpoint and storing the identified key-value records in a storage such that the identified key-value records include pointers modified to exclude ones of the plurality of key-value records that are not relevant to the checkpoint. The computing system can then recover the skip list based on the created checkpoint.

Skip-list checkpoint creation

Techniques are disclosed relating to a database recovery routine to start up a database system in response to a database failure. The database system accesses checkpoint information identifying a set of active database transactions that were active at a flush point that occurred before the database failure. As a part of the database recovery routine, the database system replays database transactions that occurred between a recovery point and the flush point. The database transactions include the set of active database transactions but exclude any committed or aborted database transactions that occurred between the recovery point and the flush point such that less than a total number of database transactions occurring between the recovery point and the flush point are replayed. The database system further replays, without excluding committed or aborted database transactions, database transactions occurring between the flush point and a recovery end point at which the database failure occurred.

Redo avoidance during database recovery

The disclosed techniques reduce a responsiveness time for a secondary node state of a database in switching from a second computing node to replace a first computing node acting in a primary node state, with both computing nodes performing the same database queries. The second node receives information regarding queries performed by the first node while in the primary state. In some embodiments, the second node retrieves, from a transaction log, log records detailing operations performed for database transactions. In some embodiments, the second node inserts, based on the log records, data records of the transactions into an in-memory cache of the second node that stores chains of database records from different transactions. Upon receiving sufficient information to switch to the primary state, the second node changes a mode of operation during failover making a committed transaction available for reads by subsequent database queries prior to record reordering.

Early database transaction visibility

This disclosure provides various techniques that may allow for accessing values stored in a data structure that stores multiple values corresponding to database transactions using a skip list. A key may be used to traverse the skip list to access data associated with the key. The skip list maintains on ordering of multiple keys, each associated with a particular record in the data structure, using indirect links between data records in the data structure that reference buckets included in hash table. Each bucket includes pointers to one or more records in the skip list.

Key-value storage using a skip list

Techniques are disclosed relating to restarting a database node. A database node may allocate memory segments that include a restart segment for storing data records. The database node may spawn processes to read a log and replay log records of the log to update the restart segment to store data records. The database node may determine to perform a restart operation to transition from a first mode to a second mode. Performing the restart operation may include ceasing reading the log at a stop position and storing, based on the stop position, database state information that enables the processes to resume reading the log from the stop position. The database node may further deallocate the memory segments except for the restart segment and terminate the processes. After performing the restart operation, the database node may spawn the processes, which may resume reading the log based on the database state information.

Database node soft restart

Techniques are disclosed relating to lock wait tracing. A computer system may operate a database that includes a lockable object. The computer may maintain a lock wait queue that stores an indication of processes waiting to acquire a lock on the lockable object. The computer system may store trace records for those processes that, upon releasing a respective lock on the lockable object when another process is waiting in the lock wait queue, have been in contention for the respective lock for over a threshold amount of time. The computer system may present ones of the trace records that identify a timeline that is usable to determine a set of processes that contributed to a delay in a process acquiring a lock on the lockable object.

Lock wait tracing

Techniques are disclosed relating to efficiently managing skip list data structures. In some embodiments, a computing system stores a skip list including a plurality of key-value records that include one or more pointers to others of the plurality of key-value records. The computing system scans the plurality of key-value records in key order to identify key-value records to be purged from the skip list. The scanning includes maintaining a list of key-value records that include pointers that point to key-value records that have not yet been scanned by the scanning. In response to identifying a key-value record for purging, the computing system purges the key-value record by substituting the pointers included the key-value records of the list with pointers included in the key-value record being purged.

Fast skip list purge

Systems and methods provide multi-version concurrency control of database records with uncommitted transactions. The system and methods may include receiving a query regarding a transaction counter number. When a transaction header identifying data is updated with the assigned transaction counter number, the updated transaction header may be used in identifying data as an instance of multiversion concurrency control information by the received query. A key lookup may be performed, and when the key lookup encounters an uncommitted row for a transaction number, a corresponding transaction header identifying data to identify a data array element to determine whether the transaction number is committed, and determining the transaction counter number when the transaction number is committed. The transaction counter number may be stamped on the row version record when it is determined that the row is committed.

Multiversion concurrency control of database records with uncommitted transactions

Techniques are disclosed relating to efficiently processing of concurrent database transactions. In one embodiment, a database system receives a first key-value pair for a database transaction and stores the key-value pair in a data structure for active database transactions. The storing may include indexing into a hash table of the data structure with a key of the key-value pair to identify a hash bucket of the hash table corresponding to the key, acquiring a latch associated with the identified hash bucket, and, based on a state of the acquired latch, appending, to the hash bucket, a record specifying the key-value pair. The database system may cause the key-value pair from the data structure to be committed to persistent storage in response to the database transaction being committed.

Database concurrency control through hash-bucket latching

This disclosure provides various techniques that may allow for key-level lock requests in a database to be managed by code that manages a memory buffer for the database. The transaction-level locks and other higher level locks may be managed by a separate lock manager. In an embodiment, the code inserts key-level lock requests in the record chain even if there is a conflict, and the key-level lock requests remain in the chain after the locks are subsequently granted. Preemption may also be supported in the record chain, in some embodiments.

Delegated key-level locking for a transactional multi-version key-value store

Systems and methods are provided for a shared memory between a database system and a storage catalog server, which stores system catalog overrides and locks for at least a first workflow. The current system catalog overrides include at least a portion of the system catalog overrides and locks that are retrieved by the shared memory from the storage catalog server at a predetermined point in time. The database system performs a check of at least one of index validity and tenant validity for a second workflow or other database operation by checking a state of the corresponding tenant or index in the current system catalog overrides. The system catalog overrides suppress visibility of the respective system catalog changes when the database system determines that at least one of the index and the tenant is invalid based on the current system catalog overrides for the first workflow.

Systems and methods of operation lock management and system catalog overrides in database systems

A log structured content addressable deduplicated data storage system may be used to store deduplicated data. Data to be stored is partitioned into data segments. Each unique data segment is associated with a label. The storage system maintains a transaction log. Mutating storage operations are initiated by storing transaction records in the transaction log. Additional transaction records are stored in the log when storage operations are completed. Upon restarting an embodiment of the data storage system, the transaction records from the transaction logs are replayed to recreate the state of the data storage system. The data storage system updates file system metadata with transaction information while a storage operation associated with the file is being processed. This transaction information serves as atomically updated transaction commit points, allowing fully internally consistent snapshots of deduplicated volumes to be taken at any time.

Growing community of inventors

San Francisco, CA, United States of America

James E Mace