Vmware, Inc.

Emc Corporation

Vmwre, Inc.

Asit Desai

Prasanna Aithal

Prasad Rao Jangam

Bryan Branstetter

Pradeep H Krishnamurthy

Mahesh Shivanagouda Hiregoudar

Eric Shannon Knauft

Srinivasa Shantharam

Rohan Pasalkar

Wenguang Wang

Enning Xiang

Richard C Zulch

Alexander T Garthwaite

Varun A Shah

Sunil Satnur

Derek Uluski

Ilya Languev

Raghavan Pichai

Yiqi Xu

Abhay Kumar Jain

Ojan Thornycroft

Antony E Boggis

Maxime Austruy

Jad Chamcham

Christian Czezatke

Christian Matthew Leroy

Sriram Patil

Raveesh Ahuja

Anoop Jaishankar

Vamsi Gunturu

Ye Zhang

Tao Xie

Ruiling Dou

Zongliang Li

Peng Dai

Dennis Bishop Jones

Vivek Patidar

Tan Li

Wangping He

Gourav Kumar Sakargayan

Yawen Chien

Huseyin Sular

Xiaoou Zhao

Litao Xia

The disclosure herein describes converting a disk cluster to a different format. A format conversion instruction associated with a disk cluster is received. A first subgroup of disks of the disk cluster that are the emptiest disks of the disk cluster are identified and all data is evacuated from the first subgroup of disks to other disks of the disk cluster. The first subgroup of disks is reformatted based on the received format conversion instruction. A group of data objects stored in the disk cluster is converted based on the format conversion instruction and the converted group of data objects are written to the reformatted first subgroup of disks. The process iterates through the disks of the disk cluster to reformat all disks and convert all data objects based on the received format conversion instruction. The process reduces the write operations required to convert the format of the disk cluster.

Converting the format of a distributed object storage with reduced write operations

Example methods and systems for accessing data in a log-structured file system having a plurality of snapshots of storage objects backed by a first-level copy-on-write (COW) B+ tree data structure and a plurality of second-level B+ tree data structures have been disclosed. One example method includes obtaining a first first-level mapping associated with a first snapshot from the plurality of snapshots based on a first logical block address, wherein each of the plurality of snapshots corresponds to each of the plurality of second-level B+ tree data structures, identifying a first second-level B+ tree data structure corresponding to one of the plurality of snapshots based on the first first-level mapping, obtaining a first second-level mapping based on the first logical block address in the first second-level B+ tree data structure, obtaining a first physical block address based on the first second-level mapping, and accessing data at the first physical block address.

Two-level logical to physical mapping mechanism in a log-structured file system

An example method of upgrading a distributed storage object from a first version to a second version includes: querying metadata of a first component configured according to the first version of the distributed storage object, the metadata defining extents of data on a disk group of the first component; populating, for a second component configured according to the second version of the distributed storage object, logical and middle maps based on the metadata such that initial entries in the logical map point to initial entries in the middle map, and the initial entries in the middle map point to physical addresses of the disk group of the first component; and reading the data from the disk group of the first component and writing the data to a disk group of the second component while updating the initial entries in the middle map.

Upgrading the file system of objects in a distributed storage system

A method for dynamic storage object configuration in a datacenter is provided. Embodiments include determining a number of fault domains in a storage cluster that have sufficient storage capacity for creating a storage object. Embodiments include applying a dynamic fault tolerance policy to the number of fault domains that have sufficient capacity for creating the storage object in order to determine a number of host failures to tolerate for the storage object, the dynamic fault tolerance policy specifying a manner of determining, for any respective storage object, a respective number of host failures to tolerate for storing the respective storage object in a respective storage cluster based on at least a respective number of fault domains of the respective storage cluster. Embodiments include implementing the storage object on the storage cluster based on the number of host failures to tolerate for the storage object.

Dynamic object policy reconfiguration mechanism for object storage system

The disclosure herein describes enhancing data durability of a base component of a data object using an unplanned delta component during transient fault unavailability. A base component of a data object becoming unavailable due to a transient fault is detected. A delta component associated with the base component is generated, wherein the delta component includes unwritten storage space with an address space and a tracking bitmap including a plurality of bits associated with data blocks of the address space of the delta component. The stale LSN with which the base component is associated is assigned to the delta component and the delta component is synchronized with an active component of the data object based on the assigned stale LSN. The delta component records write I/O targeted for the base component and, based on detecting the base component becoming available, the base component is synchronized with the delta component.

Maintaining durability of a data object using unplanned delta components during transient failures

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for maintaining fault tolerance for new writes in a storage system when one or more components of the storage system are unavailable. One of the methods includes determining that one or more first disks of a capacity object of a storage system are unavailable, wherein the storage system comprises a segment usage table identifying the plurality of segments of the capacity object; in response: identifying a plurality of available second disks, adding a plurality of new segments corresponding to the second disks to the capacity object, and adding data identifying the plurality of new segments to the segment usage table; and for each of one or more new write requests to the capacity object: identifying an available segment from the plurality of new segments, and writing data associated with the new write request to the identified available segment.

Systems and methods of maintaining fault tolerance for new writes in degraded erasure coded distributed storage

Techniques for combining the metadata and data address spaces of a distributed storage object are provided. In one set of embodiments, a distributed storage system can receive a request to provision a storage object. In response, the distributed storage system can create, in accordance with an erasure coding scheme, one or more capacity components for holding data of the storage object; create, in accordance with a mirroring scheme having an equivalent level of fault tolerance as the erasure coding scheme, one or more metadata components for holding metadata of the storage object; and create a composite object configuration tree for the storage object that includes first and second subtrees, where the first subtree comprises an indication of the mirroring scheme and references to the one or more metadata components, and where the second subtree comprises an indication of the erasure coding scheme and references to the one or more capacity components.

Combining the metadata and data address spaces of a distributed storage object via a composite object configuration tree

Techniques for improving the read performance of an LFS-based storage system that supports COW snapshotting are provided. In one set of embodiments, the storage system can implement an intermediate map for each storage object in the system that is keyed by a composite key consisting of snapshot identifier (major key) and LBA (minor key). With this approach, contiguous logical block addresses (LBAs) of a storage object or its snapshots will map to contiguous &#x3c;Snapshot ID, LBA&#x3e;-to-PBA mappings in the storage object's intermediate map, resulting in good spatial locality for those LBAs and robust read performance.

Read performance of log-structured file system (LFS)-based storage systems that support copy-on-write (COW) snapshotting

The disclosure herein describes placing delta components of a base component in target fault domains. One or more delta components are generated. When a first fault domain that lacks a sibling component of the base component is identified, the first fault domain is selected as a single delta target fault domain and a single delta component is placed on the single delta target fault domain. When a second fault domain that includes a first sibling component of the base component is identified and a third fault domain that includes a second sibling component of the base component is identified, the second fault domain and the third fault domain are selected as a first double delta target fault domain and a second double delta target fault domain, and a first double delta component and a second double delta component are placed on the first and second double delta target fault domains.

Identifying fault domains for delta components of a distributed data object

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for reclaiming one or more portions of storage resources in a computer system serving one or more virtual computing instances, where the storage resources in the computer system are organized in clusters of storage blocks. In one aspect, a method includes maintaining a respective block tracking value for each storage block that indicates whether a call to reclaim the storage block is outstanding; determining, from the block tracking values, a respective cluster priority value for each of the clusters based on a count of storage blocks in the respective cluster for which a call to reclaim is outstanding; and reclaiming a first portion of storage resources in the computer system in accordance with the cluster priority values.

Reclaiming storage resources

System and method for managing space in storage object structures stored in a storage system uses an object B tree structure to manage a storage object in a storage object structure in the storage system, which provides virtual physical block addresses for data of the storage object and for metadata of nodes of the object B tree structure. The metadata of the nodes of the object B tree structure is written in the storage object structure using first real physical block addresses, which are translated from the virtual block addresses for the metadata of the nodes of the object B tree structure using a metadata system file B tree structure. The data of the storage object is written in the storage object structure using second real physical block addresses, which are translated from the virtual block addresses for the data of the storage object using a data system file B tree structure.

System and method for managing space in storage object structures

System and method for creating group snapshots of multiple storage objects uses storage structures to store the storage objects in a storage system. In order to create a group snapshot of storage objects, a prepare snapshot process is executed at each of multiple host computers managing the storage objects as storage structures to block input and output operations on the storage objects stored in the storage structures. A commit snapshot process is then executed at each of the multiple host computers to create snapshots of the storage object in the respective storage structures. In addition, the snapshots of the storage object are recorded in the respective storage structures as snapshot entries in the respective storage structures.

System and method for creating group snapshots

The disclosure herein describes enhancing data durability of a base component using multiple delta components. A first and second delta component are generated based on the base component becoming unavailable. A write operation targeted for the base component is routed to the first delta component and to the second delta component. Based on routing the write operation targeted for the base component to the first delta component and to the second delta component, a bit associated with a data block affected by the write operation is changed in each of the tracking bitmaps of the first and second delta components. Based on detecting the base component becoming available, one delta component of the first and second delta components is selected, and the data block affected by the routed write operation is synchronized from the selected delta component to the base component. Later, the first and second delta components are removed.

Synchronizing a stale component of a distributed object using multiple delta components during maintenance

System and method for creating linked clones of storage objects stored in a storage uses a target storage structure that includes a linked clone being created from a selected snapshot of a storage object and surface snapshots. The linked clone is created in the target storage structure with references to the selected snapshot of the storage object in a source storage structure without completely copying the selected snapshot to the target storage structure. The surface snapshots, which correspond to other snapshots of the storage object in the source storage structure, are created in the target storage structure with references to the other snapshots without completely copying the other snapshots to the target storage structure to produce the linked clone with similar snapshots as the storage object at a point in time when the selected snapshot was captured.

System and method for creating linked clones of storage objects with surface snapshots

The systems described herein are configured to enhance the efficiency of memory usage and access in a VM file system data store with respect to allocating memory in large and small file block clusters using affinity metadata and propagating and maintaining the affinity metadata in support of the described allocation. In order to maintain affinity metadata of the large file block cluster, affinity generation values stored on the large file block cluster are read and cached affinity generation values for each small file block cluster are read from an in-memory cache associated with the large file block cluster. When the stored affinity generation values and the cached affinity generation values do not match, affinity metadata from all the small file block clusters associated with the large file block cluster is used to update the affinity metadata of the large file block cluster and the associated cache.

Propagating affinity data to large file block clusters in a file system

Growing community of inventors

San Ramon, CA, United States of America

Asit Desai