Netapp, Inc.

Network Appliance, Inc.

Data General Corporation

Emc Corporation

Jeffrey S Kimmel

Rajesh Sundaram

Blake H Lewis

Ashish Prakash

T Byron Rakitzis

Robert L Fair

Susan M Coatney

Arthur F Lent

Eric C Hamilton

Vijayan Rajan

Steve Miller

Bharat Chandra Baddepudi

David A Grunwald

Mohan Srinivasan

Stephen William Daniel

Brian J Pawlowski

Ling Zheng

Steven Robert Kleiman

Yuedong Mu

John T Lloyd

Santosh Ananth Rao

David Brittain Bolen

Brian Christopher Parkison

Edward D McClanahan

Steven C Miller

Douglas P Doucette

George Totolos, Jr

Tim K Emami

Yinfung Fong

Dhaval Patel

Manish Swaminathan

Randy Pafford

Christopher Joseph Corsi

Gary Ross

Venkatesh Babu Chitlur Srinivasa

Michael W J Hordijk

Matthew S DeBergalis

Roy E Clark

Abhijeet Prakash Gole

Miles Aram De Forest

Robert Anthony Alfieri

Alan Rowe

Kayuri Hasmukh Patel

Sandeep Yadav

Radek Aster

Garth Richard Goodson

Mark A O'Connell

Joydeep Sen Sarma

Rickard Edward Faith

Naresh M Patel

Sunitha Sunil Sankar

Peter B Everdell

Grace Ho

Darrell Gene Suggs

Anshul Pundir

James R Grier

John David Strunk

Arun Rokade

Tomislav Grcanac

Ning Zhao

Long Yang

Don Bolinger

Christopher G Peak

Suhas Prakash

Faramarz Rabii

Alexander D Petruncola

William K McGrath

Elizabeth H Reeves

Michael J McLeod

Nareshkumar Maganlal Patel

Robert Hawley

Joseph Caradonna

Sathya Bettadapura

T Bryon Rakitzis

Guy A Simpson

Subramaniam Perivagaram

David W Mitchell

Brian Pawlowski

A technique restores a file system of a storage input/output (I/O) stack to a deterministic point-in-time state in the event of failure (loss) of non-volatile random access memory (NVRAM) of a node. The technique enables restoration of the file system to a safepoint stored on storage devices, such solid state drives (SSD), of the node with minimum data and metadata loss. The safepoint is a point-in-time during execution of I/O requests (e.g., write operations) at which data and related metadata of the write operations prior to the point-in-time are safely persisted on SSD such that the metadata relating to an image of the file system on SSD (on-disk) is consistent and complete. Upon reboot after NVRAM loss, the technique identifies (i) the most recent safepoint, as well as (ii) the inflight writes that were persistently stored on disk after the most recent safepoint. The data and metadata of those inflight writes are then deleted to place the on-disk file system to its state at the most recent safepoint.

NVRAM loss handling

A technique reduces an amount of metadata stored in a memory of a node in a cluster. An extent store layer of a storage input/output (I/O) stack executing on the node stores key-value pairs in a plurality of data structures, e.g., cuckoo hash tables, resident in the memory. The cuckoo hash table embodies metadata that describes an extent and, as such, may be organized to associate a location on disk with a value that identifies the location on disk. The value may be embodied as a locator that includes a reference count used to support deduplication functionality of the extent store layer with respect to the extent. The reference count is divided into two portions: a delta count portion stored in memory for each slot of the hash table and an overflow count portion stored on disk in a header of each extent. One bit of the delta count portion is reserved as an overflow bit that indicates whether the in-memory reference count has overflowed. Another bit of the delta count portion is reserved as a sign bit that indicates whether the value of the remaining delta count portion, which stores the 'delta' of the reference count, is positive or negative. Overflow updates to the overflow count portion on disk are postponed until all of the bits of the delta count portion are consumed as negative/positive transitions.

Technique for reducing metadata stored in a memory of a node

A technique perturbs an extent key to compute a candidate extent key in the event of a collision with metadata (i.e., two extents having different data that yield identical hash values) stored in a memory of a node in a cluster. The perturbing technique may be used to compute a candidate extent key that is not previously stored in an extent store instance. The candidate extent key may be computed from a hash value of an extent using a perturbing algorithm, i.e., a hash collision computation, which illustratively adds a perturb value to the hash value. The perturb value is illustratively sufficient to ensure that the candidate extent key resolves to a same hash bucket and node (extent store instance) as the original extent key. In essence, the technique ensures that the original extent key is perturbed in a deterministic manner to generate the candidate extent key, so that the original extent and candidate extent key 'decode' to the same hash bucket and extent store instance.

Perturb key technique

A hybrid message-based scheduling technique efficiently load balances a storage I/O stack partitioned into one or more non-blocking (i.e., free-running) messaging kernel (MK) threads that execute non-blocking message handlers (i.e., non-blocking services) and one or more operating system kernel blocking threads that execute blocking services. The technique combines the blocking and non-blocking services within a single coherent extended programming environment. The messaging kernel (MK) operates on processors apart from the operating system kernel that are allocated from a predetermined number of logical processors (i.e., hyper-threads) for use by an MK scheduler to schedule the non-blocking services within storage I/O stack as well as allocate a remaining number of logical processors for use by the blocking services. In addition, the technique provides a variation on a synchronization primitive that allows signaling between the two types of services (i.e., non-blocking and blocking) within the extended programming environment.

Hybrid message-based scheduling technique

A cache affinity and processor utilization technique efficiently load balances work in a storage input/output (I/O) stack among a plurality of processors and associated processor cores of a node. The storage I/O stack employs one or more non-blocking messaging kernel (MK) threads that execute non-blocking message handlers (i.e., non-blocking services). The technique load balances work between the processor cores sharing a last level cache (LLC) (i.e., intra-LLC processor load balancing), and load balances work between the processors having separate LLCs (i.e., inter-LLC processor load balancing). The technique may allocate a predetermined number of logical processors for use by an MK scheduler to schedule the non-blocking services within the storage I/O stack, as well as allocate a remaining number of logical processors for use by blocking services, e.g., scheduled by an operating system kernel scheduler.

Cache affinity and processor utilization technique

Data consistency and availability can be provided at the granularity of logical storage objects in storage solutions that use storage virtualization in clustered storage environments. To ensure consistency of data across different storage elements, synchronization is performed across the different storage elements. Changes to data are synchronized across storage elements in different clusters by propagating the changes from a primary logical storage object to a secondary logical storage object. To satisfy the strictest RPOs while maintaining performance, change requests are intercepted prior to being sent to a filesystem that hosts the primary logical storage object and propagated to a different managing storage element associated with the secondary logical storage object.

Granular sync/semi-sync architecture

A flash-optimized, log-structured layer of a file system of a storage input/output (I/O) stack executes on one or more nodes of a cluster. The log-structured layer of the file system provides sequential storage of data and metadata (i.e., a log-structured layout) on solid state drives (SSDs) of storage arrays in the cluster to reduce write amplification, while leveraging variable compression and variable length data features of the storage I/O stack. The data may be organized as an arbitrary number of variable-length extents of one or more host-visible logical units (LUNs) served by the nodes. The metadata may include mappings from host-visible logical block address ranges (i.e., offset ranges) of a LUN to extent keys, as well as mappings of the extent keys to SSD storage locations of the extents. The storage location of an extent on SSD is effectively 'virtualized' by its mapped extent key (i.e., extent store layer mappings) such that relocation of the extent on SSD does require update to volume layer metadata (i.e., the extent key sufficiently identifies the extent).

Flash optimized, log-structured layer of a file system

Embodiments described herein are directed to a file system driven RAID rebuild technique. A layered file system may organize storage of data as segments spanning one or more sets of storage devices, such as solid state drives (SSDs), of a storage array, wherein each set of SSDs may form a RAID group configured to provide data redundancy for a segment. The file system may then drive (i.e., initiate) rebuild of a RAID configuration of the SSDs on a segment-by-segment basis in response to cleaning of the segment (i.e., segment cleaning). Each segment may include one or more RAID stripes that provide a level of data redundancy (e.g., single parity RAID 5 or double parity RAID 6) as well as RAID organization (i.e., distribution of data and parity) for the segment. Notably, the level of data redundancy and RAID organization may differ among the segments of the array.

File system driven raid rebuild technique

A consistency group is used as a basic unit of data management of storage containers served by a storage input/output (I/O) stack executing on one or more nodes of a cluster. The storage container may be a LUN embodied as parent volume (active volume), a snapshot (represented as an independent volume embodied as read-only copy of the active volume), and a clone (represented as another independent volume embodied as a read-write copy (clone) of the active volume). A consistency group (CG) is a set (i.e., collection) of objects, e.g., LUNs or other CGs (nested CG), which may be managed and operated upon collectively by an administrative command via a Storage Area Network administration layer (SAL) of the storage I/O stack. The SAL may interact with one or more layers of the storage I/O stack to (i) create a clone of a set of object members of the CG; (ii) create one or more snapshots of the set of object members of the CG; (iii) restore the set of object members of the CG from a group of CG snapshots; (iv) replicate the set of object members of the CG as a single entity; and (v) delete a CG and a nested CG according to specific semantics.

Consistency group management

A technique efficiently creates a snapshot for a logical unit (LUN) served by a storage input/output (I/O) stack executing on a node of a cluster that organizes data as extents referenced by keys. In addition, the technique efficiently creates one or more snapshots for a group of LUNs organized as a consistency group (CG) and served by storage I/O stacks executing on a plurality of nodes of the cluster. To that end, the technique involves a plurality of indivisible operations (i.e., transactions) of a snapshot creation workflow administered by a Storage Area Network (SAN) administration layer (SAL) of the storage I/O stack in response to a snapshot create request issued by a host. The SAL administers the snapshot creation workflow by initiating a set of transactions that includes, inter alia, (i) installation of barriers for LUNs (volumes) across all nodes in the cluster that participate in snapshot creation, (ii) creation of point-in-time (PIT) markers to record those I/O requests that are included in the snapshot, and (iii) updating of records (entries) in snapshot and volume tables of a cluster database (CDB).

Snapshot creation workflow

In one embodiment, a node coupled to solid state drives (SSDs) of a plurality of storage arrays executes a storage input/output (I/O) stack having a plurality of layers. The node includes a non-volatile random access memory (NVRAM). A first portion of the NVRAM is configured as a write-back cache to store write data associated with a write request and a second portion of the NVRAM is configured as one or more non-volatile logs (NVLogs) to record metadata associated with the write request. The write data is passed from the write-back cache over a first path of the storage I/O stack for storage on a first storage array and the metadata is passed from the one or more NVLogs over a second path of the storage I/O stack for storage on a second storage array, wherein the first path is different from the second path.

NVRAM caching and logging in a storage system

The embodiments described herein are directed to the use of hashing in a file system metadata arrangement that reduces an amount of metadata stored in a memory of a node in a cluster and that reduces the amount of metadata needed to process an input/output (I/O) request at the node. Illustratively, the embodiments are directed to cuckoo hashing and, in particular, to a manner in which cuckoo hashing may be modified and applied to construct the file system metadata arrangement. In an embodiment, the file system metadata arrangement may be illustratively include a hash collision technique that employs a hash collision computation to determine a unique candidate extent key (having a candidate hash table index) in the event of a collision, i.e., a hash table index collides with a slot of a hash table matching a key found in the slot.

Set-associative hash table organization for efficient storage and retrieval of data in a storage system

Growing community of inventors

Chapel Hill, NC, United States of America

Jeffrey S Kimmel