Vmware, Inc.

Robert T Johnson

Richard Paul Spillane

Abhishek Gupta

Kapil Chowksey

Wenguang Wang

Jorge Guerra Delgado

Srinath Premachandran

Sandeep Rangaswamy

Christos Karamanolis

Junlong Gao

Maxime Austruy

Ittai Abraham

Mark E Liffring

Mark S Shander

Christopher M Severns

Margaret Douglas

Ty A Larsen

Zachary Wade Taylor

Daniel M Mitchell

Aishwarya Ganesan

Ramnatthan Alagappan

Alexander John Horton Conway

Yi Xu

Thomas R Hasenoehrl

Todd B Brouwer

Carolyn Kerr

Arnold W Nordsieck

Timothy E Jackson

Robert L Springgay

John T Peters

Steven M Walstrom

A method of writing to a tiered memory system of a computing device, the tiered memory system including volatile memory and persistent memory (PMEM), includes the steps of: in response to a first write request including first data to write to a first page of the tiered memory system, copying contents of the first page to a second page located in the PMEM; after copying the contents of the first page to the second page, writing the first data to the second page; and after writing the first data to the second page, updating a first mapping of the tiered memory system to reference the second page instead of the first page.

Crash-safe tiered memory system

The disclosure herein describes writing data to a log-structured merge (LSM) tree file system on an object storage platform. Write data instructions indicating data for writing to the LSM tree file system are received. Based on the received instructions, the data is written to the first data cache. Based on an instruction to transfer data in the live data cache to the LSM tree file system, the first data cache is converted to a stable cache. A second data cache configured as a live data cache is then generated based on cloning the first data cache. The data in the first data cache is then written to the LSM tree file system. Use of a stable cache and a cloned live data cache enables parallel writing data to the file system by the stable cache and handling write data instructions by the live data cache.

Writing data to an LSM tree file structure using consistent cache staging

Embodiments described herein are related to a method of scanning a B-tree. For example, a method comprises receiving a scan request to scan a B-tree having a plurality of levels, each level comprising one or more nodes, wherein for each of one or more levels of the plurality of levels, nodes are grouped into groups, where nodes of any given group are stored across sequential disk blocks. The method further comprises generating a queue for each level of the B-tree. For each queue, the method further comprises loading into memory a next group of nodes based upon determining a storage location of a node of the next group of nodes.

Read iterator for pre-fetching nodes of a B-tree into memory

The disclosure herein describes synchronizing a data cache and an LSM tree file system on an object storage platform. Instructions to send a cached data set from the data cache to the LSM tree file system are received. An updated metadata catalog is generated. If the LSM tree structure is out of shape, compaction is performed on the LSM tree file system which may be on a different system or server. When an unmerged compacted metadata catalog is identified, a merged metadata catalog is generated, based on the compacted metadata catalog and the cached data set, and associated with the cached data set. The cached data set and the associated metadata catalog are sent to the LSM tree file system, whereby the data cache and the LSM tree file system are synchronized. Synchronization is enabled without the data cache or file system being locked and/or waiting for the other entity.

Lockless synchronization of LSM tree metadata in a distributed system

A B-tree associated with a file system on a storage volume includes a hierarchy of nodes. Each node includes a buffer portion to store key-value pairs as messages in the buffer. Each node can be characterized by having a maximum allowable size that is periodically updated at run time. The buffers in the nodes of the B-tree are therefore characterized by having a maximum allowed size that can vary over time.

Auto-tuned write-optimized key-value store

Embodiments herein are directed towards systems and methods for performing range lookups in B-trees. One example method involves receiving a request to return key-value pairs within a range of keys from the B-tree. The B-tree includes a plurality of nodes, each node being associated with a buffer that stores key-value pairs. The method further involves determining a fractional size of the range of keys. The method further involves, for each level of the B-tree, obtaining from within one or more buffers of one or more nodes of the level, a set of key-value pairs within the range of keys up to a size equal to the fractional size and transferring the set of key-value pairs to a result data structure. The method further involves sorting and merging all key-value pairs in the result data structure and returning the result data structure in response to the request.

Large range lookups for B-tree

The disclosure herein describes providing and accessing data on an object storage platform using a log-structured merge (LSM) tree file system. The LSM tree file system on the object storage platform includes sorted data tables, each sorted data table including a payload portion and an index portion. Data is written to the LSM tree file system in at least one new sorted data table. Data is ready by identifying a data location of the data based on index portions of the sorted data tables and reading the data from a sorted data table associated with the identified data location. The use of the LSM tree file system on the object storage platform provides an efficient means for interacting with the data stored thereon.

Using an LSM tree file structure for the on-disk format of an object storage platform

The disclosure herein describes synchronizing cached index copies at a first site with indexes of a log-structured merge (LSM) tree file system on an object storage platform at a second site. An indication that the LSM tree file system has been compacted based on a compaction process is received. A cached metadata catalog of the included parent catalog version at the first site is accessed. A set of cached index copies is identified at the first site based on the metadata of the cached metadata catalog. The compaction process is applied to the identified set of cached index copies and a compacted set of cached index copies is generated at the first site, whereby the compacted set of cached index copies is synchronized with a respective set of indexes of the plurality of sorted data tables of the LSM tree file system at the second site.

Synchronization of index copies in an LSM tree file system

A buffer tree structure includes, at each internal node, a buffer having a compacted portion and an uncompacted portion. Insertion of data elements to the buffer tree can occur units called packets. A packet is initially stored in the uncompacted portion of a receiving node's buffer. When a compaction trigger condition exists, packet compaction is performed including a data element compaction operation. A buffer-emptying (flush) operation pushes the compacted packets to children nodes.

Bounding cost of flushes in buffer trees

Embodiments described herein are related to bulk loading data into a B-tree. Embodiments include generating a first leaf node of a B-tree by allocating a first page for the first leaf node from a leaf page queue comprising a first plurality of sequential pages; and writing one or more tuples to the first page allocated for the first leaf node. Embodiments further include generating an parent node for the first leaf node and a second leaf node of the B-tree by allocating a third page for the parent node from an parent page queue comprising a second plurality of sequential pages, the parent node comprising a first indication of the first leaf node and a second indication of the second leaf node, the first indication and the second indication stored in the third page allocated for the parent.

Bulk-load for B-trees

A buffer tree structure includes, at each internal node, a buffer having a compacted portion and an uncompacted portion. Insertion of data elements to the buffer tree can occur units called packets. A packet is initially stored in the uncompacted portion of a receiving node's buffer. After a time, packets in the uncompacted portion of a buffer are combined into compacted packets in the compacted portion of the buffer. A buffer-emptying (flush) operation pushes the compacted packets to children nodes.

Reducing write amplification in buffer trees

Certain aspects provide systems and methods for performing an operation on a B-tree. A method comprises writing a message associated with the operation to a first slot in a first buffer of a first non-leaf node of the B-tree in an append-only manner, wherein a first filter associated with the first slot is used for query operations associated with the first slot. The method further comprises determining that the first buffer is full and, upon determining to flush the message to a non-leaf child node, flushing the message in an append-only manner to a second slot in a second buffer of the non-leaf child node, wherein a second filter associated with the second slot is used for query operations associated with the second slot. The method further comprises, upon determining to flush the message to a leaf node, flushing the message to the leaf node in a sorted manner.

Trading off cache space and write amplification for B-trees

Certain aspects provide systems and methods of compacting data within a log-structured merge tree (LSM tree) using sharding. In certain aspects, a method includes determining a size of the LSM tree, determining a compaction time for a compaction of the LSM tree based on the size, determining a number of compaction entities for performing the compaction in parallel based on the compaction time, determining a number of shards based on the number of compaction entities, and determining a key range associated with the LSM tree. The method further comprises dividing the key range by the number of shards into a number of sub key ranges, wherein each of the number of sub key ranges corresponds to a shard of the number of shards and assigning the number of shards to the number of compaction entities for compaction.

Systems and methods for performing scalable Log-Structured Merge (LSM) tree compaction using sharding

Balancing write amplification and space amplification in buffer trees

A B-tree associated with a file system on a storage volume includes a hierarchy of nodes. Each node includes a buffer portion that can be characterized by a fixed maximum allowable size to store key-value pairs as messages in the buffer. Messages can be initially buffered in the root node of the B-tree, and flushed to descendent children from the root node. Messages stored in the buffers can be indexed using a B-tree data structure. As the B-tree data structure in a buffer grows (due to receiving flushed messages) and shrinks (due to messages being flushed), disk blocks can be allocated from the storage volume to increase the actual size of the buffer and deallocated from the buffer to reduce the actual size of the buffer.

Growing community of inventors

Palo Alto, CA, United States of America

Robert T Johnson