Netapp, Inc.

Network Appliance, Inc.

Vmware, Inc.

Blake H Lewis

Ling Zheng

John K Edwards

Subramaniam Periyagaram

Jeffrey S Kimmel

Kayuri Hasmukh Patel

Sandeep Yadav

Robert M English

Ashish Prakash

Rickard Edward Faith

Rajesh Sundaram

Peter Frank Corbett

Eric C Hamilton

Stephen William Daniel

Stephen Lee Manley

Douglas P Doucette

Vijay Mohan Deshmukh

David Hitz

Craig Anthony Johnston

Rohit Shankar Singh

Steven S Watanabe

Vinod Kumar Daga

Nikhil Mattankot

Rajesh Khandelwal

Jayalakshmi Pattabiraman

James Robert Morefield

Satya Suresh Chouta Naga Veera

Deekshith Belchapada

Ananthan Subramanian

Ram Kesavan

Srinivasan Viswanathan

Raymond C Chen

Andy Kahn

Edward Ramon Zayas

Daniel Ting

Rajesh Desai

Janice D'Sa

Ray Chen

Kapil Kumar

Wenguang Wang

Mohan Srinivasan

Eric Shannon Knauft

Vamsi Gunturu

Pascal Renauld

Vijayan Rajan

Michael Lee Federwisch

David Brittain Bolen

Brian J Pawlowski

Matt Amdur

Subramaniam Perivagaram

Jeff Kimmel

Steve Daniel

Peng Mi

John K Edwars

Brian Pawlowski

The present disclosure is related to methods, systems, and machine-readable media for log-structured file system management operations. An aggregate amount of over-provisioned computing resources assigned to a plurality of log-structured file system (LFS) objects that are allocated for performance of memory management operations associated with a virtual storage area network (vSAN) can be determined. A subset of LFS objects that are candidates for performance of a particular memory management operation based on an amount of over-provisioned computing resources consumed by one or more LFS objects among the plurality of LFS objects exceeding a resource consumption threshold associated with the determined amount of over-provisioned computing resources assigned to the plurality of LFS objects can be selected. The particular memory management operation using one or more of the candidate LFS objects can be performed.

Dynamic over-provisioning of space in a log-structured file system

A first plurality of block identifiers is sorted based, at least in part, on a measure of spatial locality. A second plurality of block identifiers is sorted based, at least in part, on the measure of spatial locality. At least the first plurality of block identifiers and the second plurality of block identifiers are incrementally merged into a third plurality of block identifiers based, at least in part, on the measure of spatial locality. A block of data corresponding to metadata associated with a plurality of block identifiers of the third plurality of block identifiers is updated.

Efficient modification of storage system metadata

A technique to name data is disclosed to allow preservation of storage efficiency over a link between a source and a destination in a replication relationship as well as in storage at the destination. The technique allows the source to send named data to the destination once and refer to it by name multiple times in the future, without having to resend the data. The technique also allows the transmission of data extents to be decoupled from the logical containers that refer to the data extents. Additionally, the technique allows a replication system to accommodate different extent sizes between replication source and destination while preserving storage efficiency.

Handling data extent size asymmetry during logical replication in a storage system

A technique preserves efficiency for replication of data between a source node of a source cluster ('source') and a destination node of a destination cluster ('destination') of a clustered network. Replication in the clustered network may be effected by leveraging global in-line deduplication at the source to identify and avoid copying duplicate data from the source to the destination. To ensure that the copy of the data on the destination is synchronized with the data received at the source, the source creates a snapshot of the data for use as a baseline copy at the destination. Thereafter, new data received at the source that differs from the baseline snapshot are transmitted and copied to the destination. In addition, the source and destination nodes negotiate to establish a mapping of name-to-data when transferring data (i.e., an extent) between the clusters. Illustratively, the name is an extent key for the extent, such that the negotiated mapping established by the source and destination is based on the extent key associated with the extent.

Technique for preserving efficiency for replication between clusters of a network

A N-way merge technique efficiently updates metadata in accordance with a N-way merge operation managed by a volume layer of a storage input/output (I/O) stack executing on one or more nodes of a cluster. The metadata is embodied as mappings from logical block addresses (LBAs) of a logical unit (LUN) accessible by a host to durable extent keys, and is organized as a multi-level dense tree. The mappings are organized such that a higher level of the dense tree contains more recent mappings than a next lower level, i.e., the level immediately below. The N-way merge operation is an efficient (i.e., optimized) way of updating the volume metadata mappings of the dense tree by merging the mapping content of all three levels in a single iteration, as opposed to merging the content of the first level with the content of the second level in a first iteration of a two-way merge operation and then merging the results of the first iteration with the content of the third level in a second iteration of the operation.

N-way merge technique for updating volume metadata in a storage I/O stack

A flash-optimized, log-structured layer of a file system of a storage input/output (I/O) stack executes on one or more nodes of a cluster. The log-structured layer of the file system provides sequential storage of data and metadata (i.e., a log-structured layout) on solid state drives (SSDs) of storage arrays in the cluster to reduce write amplification, while leveraging variable compression and variable length data features of the storage I/O stack. The data may be organized as an arbitrary number of variable-length extents of one or more host-visible logical units (LUNs) served by the nodes. The metadata may include mappings from host-visible logical block address ranges (i.e., offset ranges) of a LUN to extent keys, as well as mappings of the extent keys to SSD storage locations of the extents. The storage location of an extent on SSD is effectively 'virtualized' by its mapped extent key (i.e., extent store layer mappings) such that relocation of the extent on SSD does require update to volume layer metadata (i.e., the extent key sufficiently identifies the extent).

Flash optimized, log-structured layer of a file system

A system can maintain multiple queues for deduplication requests of different priorities. The system can also designate priority of storage units. The scheduling priority of a deduplication request is based on the priority of the storage unit indicated in the deduplication request and a trigger for the deduplication request.

Scheduling deduplication in a storage system

The embodiments described herein are directed to an organization of metadata managed by a volume layer of a storage input/output (I/O) stack executing on one or more nodes of a cluster. The metadata managed by the volume layer, i.e., the volume metadata, is illustratively embodied as mappings from addresses, i.e., logical block addresses (LBAs), of a logical unit (LUN) accessible by a host to durable extent keys maintained by an extent store layer of the storage I/O stack. In an embodiment, the volume layer organizes the volume metadata as a mapping data structure, i.e., a dense tree metadata structure, which represents successive points in time to enable efficient access to the metadata.

Dense tree volume metadata organization

In one embodiment, a layered file system includes a volume layer and an extent store layer configured to provide sequential log-structured layout of data and metadata on solid state drives (SSDs) of one or more storage arrays. The data is organized as variable-length extents of one or more logical units (LUNs). The metadata includes volume metadata mappings from offset ranges of a LUN to extent keys and extent metadata mappings of the extent keys to storage locations of the extents on the SSDs. The extent store layer maintaining the extent metadata mappings determines whether an extent is stored on a storage array, and, in response to determination that the extent is stored on the storage array, returns an extent key for the stored extent to the volume layer to enable global inline de-duplication that obviates writing a duplicate copy of the extent on the storage array.

Global in-line extent-based deduplication

Partially overwriting a compression group without decompressing compressed data can consumption of resources for the decompression. A storage server partially overwrites the compression group when a file block identifier of a client's write request resolves to the compression group. The compression group remains compressed while the partial overwriting is performed.

Overwriting part of compressed data without decompressing on-disk compressed data

Growing community of inventors

Palo Alto, CA, United States of America

Blake H Lewis