Vmware, Inc.

Vwware, Inc.

Eric Shannon Knauft

Pascal Renauld

Wenguang Wang

Enning Xiang

Yiqi Xu

Vamsi Gunturu

Christos Karamanolis

Asit Desai

William John Earl

Ojan Thornycroft

Jin Zhang

Junlong Gao

Varun A Shah

Xin Liang Li

Hardik Singh Negi

Vamsidhar Gunturu

Mansi Shah

Radhika Vullikanti

Xiaochuan Shen

Ruocheng Li

Pratik Shridhar Desai

Yizhou Luo

Srinath Premachandran

Abhay Kumar Jain

Christian Dickmann

Gopala Suryanarayana

Haoran Zheng

Zhihao Yao

Bryan Fink

Vishal Kher

Jiaqi Zuo

Blake H Lewis

Richard Paul Spillane

Maxime Austruy

Jorge Guerra Delgado

Matthew Benjamin Amdur

Sandeep Rangaswamy

Zhaohui Guo

Edward Joseph Goggin

Satish Pudi

Radu Berinde

Mounesh Badiger

Sriram Patil

Luke Lu

Long Yang

Gregory A Smith

Kevin Rayfeng Li

Matt Amdur

Huseyin Sular

Peng Mi

Quanxing Liu

Yanxing Pan

Sudheer Thokala

Example methods and systems for accessing data in a log-structured file system having a plurality of snapshots of storage objects backed by a first-level copy-on-write (COW) B+ tree data structure and a plurality of second-level B+ tree data structures have been disclosed. One example method includes obtaining a first first-level mapping associated with a first snapshot from the plurality of snapshots based on a first logical block address, wherein each of the plurality of snapshots corresponds to each of the plurality of second-level B+ tree data structures, identifying a first second-level B+ tree data structure corresponding to one of the plurality of snapshots based on the first first-level mapping, obtaining a first second-level mapping based on the first logical block address in the first second-level B+ tree data structure, obtaining a first physical block address based on the first second-level mapping, and accessing data at the first physical block address.

Two-level logical to physical mapping mechanism in a log-structured file system

System and method for managing different classes of storage input/output (I/O) requests for a two-phase commit operation in a distributed storage system assigns reserved log sequence values to each of storage I/O requests of a first class, which are added to a two-phase commit queue. The reserved log sequence values of the storage I/O requests of the first class in the two-phase commit queue are assigned to some of the storage I/O requests of the second class, which are added to the two-phase commit queue.

Two-phase commit using reserved log sequence values

A method for efficient write-back for journal truncation is provided. A method includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction associated with one or more pages in an ordered data structure and maintaining a dirty list including an entry for each page indicated by a record in the journal. Each entry in the dirty list includes a respective first log sequence number (LSN) associated with a least recent record of the plurality of records that indicates the page and a respective second LSN associated with a most recent record of the plurality of records that indicates the page. The method includes determining to truncate the journal. The method includes identifying one or more records, of the plurality of records, from the journal to write back to a disk, where the identifying is based on the dirty list.

Efficient write-back for journal truncation

The disclosure herein describes performing resynchronization ('resync') jobs in a distributed storage system based on a parallelism policy. A resync job is obtained from a queue and input/output (I/O) resources that will be used during execution of the resync job are identified. Available bandwidth slots of each I/O resource of the identified I/O resources are determined. The parallelism policy is applied to the identified I/O resources and the available bandwidth slots. Based on the application of the parallelism policy, a bottleneck resource of the I/O resources is determined and a parallel I/O value is calculated based on the available bandwidth slots of the bottleneck resource, wherein the parallel I/O value indicates a quantity of I/O tasks that can be performed in parallel. The resync job is executed using the I/O resources, the execution of the resync job including performance of I/O tasks in parallel based on the parallel I/O value.

Performing resynchronization jobs in a distributed storage system based on a parallelism policy

A method for efficient journal truncation is provided. A method for journal truncation includes maintaining a journal in a memory of a computing system including a plurality of records. Each record indicates a transaction in an ordered data structure. The method includes maintaining a truncation queue in the memory including one or more entries. Each entry in the truncation queue includes a physical on-disk offset associated with a different record of the plurality of records. The method includes determining to truncate the journal and truncating records, of the plurality of records, from the journal starting from a beginning record in the journal up to the record with the physical on-disk offset associated a least recent entry of the one or more entries in the truncation queue, where the truncating includes removing the records from the memory.

Efficient incremental journal truncation policy

A method for modifying key-value pairs of a B+ tree is provided. The method receives a request to modify a particular key-value pair. Each node of the tree has a modification number. The method traverses a path on the tree from the root node toward the particular node. The traversing includes upon reaching a parent node of the path, acquiring a shared lock on both the parent node and a child node one level below the parent node. Upon determining that the child node is the particular node, the method stores the modification number of the particular node, releases the shared lock on the particular node, compares a current modification number of the node with its stored number, and acquires an exclusive lock on the node if the numbers are the same. The method increments the current modification number of the node and modifies it while in the exclusive lock.

Enhanced locking mechanism for B+ tree data structures

Components of a distributed data object are synchronized using streamlined tracking metadata. A target component of the distributed data object is detected as it becomes available and stale. A source component that is up-to-date and that mirrors the address space of the detected target component is identified. A set of mapped address ranges and a set of unmapped address ranges of the identified source component are obtained. A mapped address range of the target component that corresponds with an unmapped address range of the source component is identified. The identified mapped address range of the target component is then synchronized with the corresponding unmapped address range of the source component. Thus, unmapped address ranges are synchronized without using tracking metadata of the source component.

Efficient unmap handling using synchronized redundant components of data objects

The disclosure describes growing a data cache using a background hash bucket growth process. A first memory portion is allocated to the data buffer of the data cache and a second memory portion is allocated to the metadata buffer of the data cache based on the cache growth instruction. The quantity of hash buckets in the hash bucket buffer is increased and the background hash bucket growth process is initiated, wherein the process is configured to rehash hash bucket entries of the hash bucket buffer in the increased quantity of hash buckets. A data entry is stored in the data buffer using the allocated first memory portion of the data cache and metadata associated with the data entry is stored using the allocated second memory portion of the metadata buffer, wherein a hash bucket entry associated with the data entry is stored in the increased quantity of hash buckets.

Dynamic growth of data caches using background processes for hash bucket growth

The disclosure herein describes enhancing data durability of a base component using a delta component. A delta component is generated based on the base component becoming unavailable. The delta component is configured to include unwritten storage space with an address space matching the base component and a tracking bitmap associated with data blocks of the address space of the delta component. Write operations targeted for the base component are routed to the delta component. Based on the routed write operations, bits associated with data blocks affected by the write operations are changed in the tracking bitmap. Based on the base component becoming available, data blocks affected by routed write operations are identified based on the tracking bitmap and the identified data blocks are synchronized from the delta component to the base component. The delta component is then removed.

Synchronizing a stale component of a distributed object using a delta component during maintenance

An example method of upgrading a distributed storage object from a first version to a second version includes: querying metadata of a first component configured according to the first version of the distributed storage object, the metadata defining extents of data on a disk group of the first component; populating, for a second component configured according to the second version of the distributed storage object, logical and middle maps based on the metadata such that initial entries in the logical map point to initial entries in the middle map, and the initial entries in the middle map point to physical addresses of the disk group of the first component; and reading the data from the disk group of the first component and writing the data to a disk group of the second component while updating the initial entries in the middle map.

Upgrading the file system of objects in a distributed storage system

System and method for managing distributed storage objects for host unavailability in a distributed storage system uses at least one of a crash indicator in a specific on-disk block and a paused object indicator for a distributed storage object to determine whether to perform data recovery for the distributed storage object. When the crash indicator is set or the paused object indicator implies that the distributed storage object is a paused object, the distributed storage object is left as a paused object without perform the data recovery for the distributed storage object. When the crash indicator is unset and the paused object indicator implies that the distributed storage object is not a paused object, the data recovery for the distributed storage object is performed.

Limiting blast radius for critical on-disk data corruptions in a distributed storage system

The disclosure herein describes storing data using a capacity data storage tier and a smaller performance data storage tier. The capacity data storage tier includes capacity data storage hardware configured to store log-structured leaf pages (LLPs), and the performance data storage tier includes performance data storage hardware. A virtual address table (VAT) includes a set of virtual address entries referencing the LLPs. A tree-structured index includes index nodes referencing the set of virtual address entries of the VAT. Data to be stored is received, and at least a first portion of metadata associated with the received data is stored in the LLPs using the VAT, and at least a second portion of metadata associated with the received data is stored in the performance data storage tier. The architecture reduces space usage of the performance data storage tier.

Storing data in a log-structured format in a two-tier storage system

The present disclosure is related to methods, systems, and machine-readable media for log-structured file system management operations. An aggregate amount of over-provisioned computing resources assigned to a plurality of log-structured file system (LFS) objects that are allocated for performance of memory management operations associated with a virtual storage area network (vSAN) can be determined. A subset of LFS objects that are candidates for performance of a particular memory management operation based on an amount of over-provisioned computing resources consumed by one or more LFS objects among the plurality of LFS objects exceeding a resource consumption threshold associated with the determined amount of over-provisioned computing resources assigned to the plurality of LFS objects can be selected. The particular memory management operation using one or more of the candidate LFS objects can be performed.

Dynamic over-provisioning of space in a log-structured file system

Embodiments of the disclosure provide techniques for partitioning a resource object into multiple resource components of a cluster of host computer nodes in a distributed resources system. The distributed resources system translates high-level policy requirements into a resource configuration that the system accommodates. The system determines an allocation based on the policy requirements and identifies resource configurations that are available. Upon selecting a resource configuration, the distributed resources system assigns the allocation and associated values to the selected configuration and publishes the new configuration to other host computer nodes in the cluster.

Translating high level requirements policies to distributed storage configurations

Hybrid synchronization using a shadow component includes detecting a first component of a plurality of mirrored components of a distributed data object becoming unavailable. The mirrored components include a delta component (a special shadow component) and a regular mirror (shadow) component. The delta component indicates a shorter history of changes to data blocks of a log-structured file system (LFS) than is indicated by the regular mirror component. During the unavailability of the first component, at least one write I/O is committed by the delta component. The commit is tracked by the delta component in a first tracking bitmap associated with the delta component. Based at least on detecting the first component becoming available, the first component is synchronized with data from the delta component, based at least on changed data blocks indicated in the first tracking bitmap.

Hybrid synchronization using a shadow component

Growing community of inventors

San Francisco, CA, United States of America

Eric Shannon Knauft