Netapp, Inc.

Xiotech Corporation

Compellent Technologies

Dell International L.l.c.

Mark David Olson

Michael H Pittelko

Michael Henry Pittelkow

Alyssa Proulx

Michael James Klemm

Anil Paul Thoppil

Lawrence E Aszmann

Wei Sun

James Philip Wittig

Jared Cantwell

Daniel David McCarthy

Christopher Cason

Forrest Trimbell

Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list may be compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.

Combined garbage collection and data integrity checking for a distributed key-value store

A system, method, and machine-readable storage medium for determining an amount of unique data in a distributed storage system are provided. In some embodiments, a combined efficiency set for a first data set stored in the distributed storage system, such as at a volume, may be generated. The first data set may include a first subset of data and a second subset of data in the distributed storage system. Additionally, a set of efficiency sets for the first subset of data may be generated. A set difference based on the combined efficiency set and the set of efficiency sets may be computed. An amount of memory used for storing unique data of the second subset of data may be estimated based on the set difference. The unique data may be present in the second subset of data but absent from the first subset of data.

Efficiency sets for determination of unique data

Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage management system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list is compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.

A technique maintains multiple copies of data served by storage nodes of a cluster during upgrade of a storage node to ensure continuous protection of the data served by the nodes. The data is logically organized as one or more volumes on storage devices of the cluster and includes metadata that describe the data of each volume. A data protection system may be configured to maintain two copies of the data in the cluster during upgrade to a storage node that is assigned to host one of the copies of the data but that is taken offline during the upgrade. As a result, a slice service of the node may become unavailable during the upgrade. In response to the unavailability of the slice service, the technique redirects replicated data targeted to the slice service to a standby slice service according to a degraded redundant metadata (DRuM) service of the cluster.

Degraded redundant metadata, DRuM, technique

A technique is configured to maintain multiple copies of data served by storage nodes of a cluster during upgrade of a storage node to ensure continuous protection of the data served by the nodes. The data is logically organized as one or more volumes on storage devices of the cluster and includes metadata that describe the data of each volume. A data protection system may be configured to maintain at least two copies of the data in the cluster during upgrade to a storage node that is assigned to host one of the copies of the data but that is taken offline during the upgrade. As a result, an original slice service of the node may be rendered unavailable during the upgrade. In response, the technique redirects replicated data targeted to the original slice service to a standby pool of slice services in accordance with a degraded redundant metadata service of the cluster. In the event the standby slice service itself subsequently becomes unavailable, another standby slice service from the standby pool is activated to receive the subsequent data. In this manner, cascading failure of secondary slice slices is handled.

Standby copies withstand cascading fails

A method of data replication from a first data storage device to a second data storage device. The method may include generating, at the first data storage device, at spaced time intervals, a plurality of snapshots for a logical data volume of the first data storage device, the logical data volume being an abstraction of data blocks from one or more physical storage devices, each snapshot identifying changes of data for at least a portion of the logical data volume since a most previous snapshot. Also at the first data storage device, the method includes generating a delta volume, the delta volume indicating changes in the data of at least a portion of the logical data volume between two non-consecutive snapshots. The method further involves replicating the delta volume to the second data storage device, and replicating the changes to the data indicated therein at the second data storage device.

Systems and methods for replication of data utilizing delta volumes

The present disclosure relates generally to a method and system for creating, replicating, and providing access to virtual snapshots of a disk storage block of a disk storage system or subsystem. In one embodiment, the present disclosure relates to a virtual snapshot accessible to local users of a local data storage device. The virtual snapshot may direct local users to a snapshot stored on computer-readable storage medium at a remote data storage site, but give the appearance as if data of the corresponding snapshot is stored locally. The virtual snapshot is replaced by replication of the snapshot from the remote data storage site to the local data storage device. Each snapshot may relate to data of a logical data volume, the logical data volume being an abstraction of data blocks from one or more physical storage devices.

Virtual snapshot system and method

A process of determining explicitly free data space in computer data storage systems with implicitly allocated data space through the use of information provided by a hosting computer system with knowledge of what space allocated is currently being used at the time of a query, is provided. In one embodiment, a File System ('FS') is asked to identify clusters no longer in use which is then mapped to physical disks as visible to an Operating System ('OS'). The physical disks are mapped to simulated/virtualized volumes presented by a storage subsystem. By using server information regarding the FS, for those pages that are no longer in use, point in time copy (&#x201c;PITC&#x201d;) pages are marked for future PITC and will not be coalesced forward, thereby saving significant storage.

Data storage space recovery system and method

A fiber channel storage area network (SAN) provides virtualized storage space for a number of servers to a number of virtual disks implemented on various virtual redundant array of inexpensive disks (RAID) devices striped across a plurality of physical disk drives. The SAN includes plural controllers and communication paths to allow for fail-safe and fail-over operation. The plural controllers can be loosely-coupled to provide n-way redundancy and have more than one independent channel for communicating with one another. In the event of a failure involving a controller or controller interface, the virtual disks that are accessed via the affected interfaces are re-mapped to another interface in order to continue to provide high data availability. In particular, those targets, or virtual disks, which are owned by a failed device (e.g., interface or controller) are identified and their ownership information within the SAN configuration tables is revised to reflect one or more alternative devices which act as back-up devices.

System and method to failover storage area network targets from one interface to another

A fiber channel storage area network (SAN) provides virtualized storage space for a number of servers to a number of virtual disks implemented on various virtual redundant array of inexpensive disks (RAID) devices striped across a plurality of physical disk drives. The SAN includes plural controllers and communication paths to allow for fail-safe and fail-over operation. The plural controllers can be loosely-coupled to provide n-way redundancy and more than one independent channel for communicating with one another. In the event of a failure involving a controller or controller interface, the virtual disks that are accessed via the affected interfaces are re-mapped to another interface in order to continue to provide high data availability. In particular a resource re-allocation routine operating on one of the controllers employs predetermined criteria against a family of potential back-up controllers in order to select appropriate back-up controllers for a particular device that fails. The criteria, for example, can include an explicit primary/secondary table to control selecting a secondary controller for any primary controller or be more complex so as to consider such statistics as data transfer loads among the interfaces, data transfer loads among the controllers, or the number of targets assigned to each interfaces.

Growing community of inventors

Bloomington, MN, United States of America

Mark David Olson