Emc IP Holding Company LLC

Emc Corporation

Michal Marko

Walter Forrester

Jean-Pierre Bono

William C Davenport

Ahsan Rashid

Miles Aram De Forest

Ye Zhang

Philippe Armangau

Ivan Bassov

Alexander S Mathews

Kumari Bijayalaxmi Nanda

Ivan Basov

Soumyadeep Sen

Karl M Owen

Rohit K Chawla

Joris Johannes Wils

Yunfei Chen

James P Ferrigno

Nushafreen Dara Palsetia

An aspect of space accounting includes providing a counter for each of a plurality of key/value pairs for storage system operations. The key uniquely identifies a logical storage unit on which operations associated with the key are performed, the value identifies an amount of data usage with respect to the operations performed on the logical storage unit, and the counter tracks a type of the operations. An aspect also includes tracking, via corresponding counters, updates to a read/write (RW) tablet associated with a Cuckoo tree. The tracking includes updating a filter to include instances of key/value pairs and the RW tablet in which the pairs are stored. An aspect further includes tracking a plurality of read-only (RO) tablets associated with the Cuckoo tree. The tracking includes updating the filter to include instances of key/value pairs and one or more of the RO tablets in which the pairs are stored.

Space accounting for data storage usage

Techniques for performing space accounting for volume families. The techniques include maintaining two counters for each volume family, including a first counter that tracks a first amount of physical space allocated to volumes in the volume family based on the number of pages written to the volumes, and a second counter that tracks a second amount of physical space owned by the volume family, and maintaining a third counter for each volume in each branch of the volume family that tracks a number of pages among a total number of pages written to the volume that were overwritten in the immediate sequential next volume in the same branch as the volume. By maintaining, for each volume family, the first counter and the second counter, and, for each volume in the volume family, the third counter, space accounting metrics can be obtained that allow data storage activities to be performed efficiently.

Mechanisms for performing accurate space accounting for volume families

A technique for managing storage space in a data storage system generates liability values on a per-family basis, with each family including files in the file system that are related to one another by snapping. Each family thus groups together files in the file system that generally share at least some blocks among one another based on snapshot activities. Distinct files that do not share blocks based on snapping are provided in separate families. The technique further generates worst-case storage liability of a version family by differentiating between writable data objects and read-only data objects, thus allowing administrators to provide spare storage and/or prepare for increases in storage requirements as writable data objects grow and differentiate.

Managing storage capacity in version families having both writable and read-only data objects

Techniques for tracking pages that are unshared among snapshot volumes and their parent volume in a volume family. The techniques include maintaining a first counter that tracks a first number ('SnapOwnedCount') of pages among a total number of pages written to a volume that were overwritten in the immediate sequential next volume in the same branch as the volume, and a second counter that tracks a second number ('UnsharedWithParentCount') of pages among the total number of pages written to the volume that were overwritten in any sequential next volume in the same branch as the volume. By maintaining, for each volume in a volume family, a first counter and a second counter for tracking the SnapOwnedCount and the UnsharedWithParentCount, respectively, data storage recovery can be performed more effectively, allowing selected volumes and/or portions thereof to be deleted while maintaining accurate space accounting information.

Efficient space accounting mechanisms for tracking unshared pages between a snapshot volume and its parent volume

A technique for managing storage space in a data storage system implements data objects in respective files. In response to an instruction to convert a set of such files from thin to thick or from thick to thin, the data storage system checks whether it has enough available storage space to accommodate the set of files if conversion were to proceed. If so, conversion is performed, and each of the set of files is converted from thin to thick or from thick to thin, in accordance with the instruction.

Converting files between thinly and thickly provisioned states

Methods and apparatus are provided for reporting space savings due to data reduction in storage systems. An exemplary method comprises establishing a leaf pointer for a plurality of allocation units of data in a storage system; for at least a first allocation unit of data matching a second allocation unit of data, replacing the leaf pointer for the first allocation unit with a deduplication pointer to the leaf pointer of the second allocation unit; determining a total number of pointers as a sum of a number of the leaf pointers and a number of the deduplication pointers; determining a difference between the total number of pointers and the number of leaf pointers; and providing a data reduction attributed to deduplication based on the difference. The allocation units of data are optionally also compressed. Data reduction attributed to the compression can optionally be determined independent of the data reduction attributed to the deduplication.

Data reduction reporting in storage systems

A technique for facilitating file system operations in a data storage system provides multiple version family databases. Each version family database identifies members of a respective version family, i.e., a respective set of files in a file system that are related by file snapping. When a snap of a file is created, the file system updates a version family database for the version family that includes the file, such that the version family database identifies both the file and the snap.

Tracking version families in a file system

Methods and apparatus are provided for reporting space savings due to pattern matching in storage systems. An exemplary method comprises, when a given allocation unit in a storage system matches one or more predefined patterns, (i) setting a corresponding pattern flag for the given allocation unit, and (ii) incrementing at least one pattern counter; generating at least one snapshot of at least a portion of a file comprising the given allocation unit; and determining a data reduction attributed to pattern matching based on the at least one pattern counter, wherein the one or more predefined patterns in the at least one snapshot are excluded from the data reduction attributed to pattern matching.

Reporting of space savings due to pattern matching in storage systems

An improved technique involves providing file system shrink capability by shrinking the size of the volume file in which an upper deck file system is stored. Along these lines, a data storage system, upon receiving a request to shrink an upper deck file system, determines a new address in the upper deck file system, based on an amount the upper deck file system is to be shrunk, to which the end-of-file (EOF) pointer of the volume file will point. The new address will have a smaller distance from the address of the file pointer than the original address to which the EOF pointer points. The data storage system then identifies nonempty data blocks of the upper deck file system that have logical addresses having a larger distance from the address of the file pointer than the second logical address.

Upper deck file system shrink for directly and thinly provisioned lower deck file system in which upper deck file system is stored in a volume file within lower deck file system where both upper deck file system and lower deck file system resides in storage processor memory

A technique for managing storage space in a data storage system generates liability values on a per-family basis, with each family including files in the file system that are related to one another by snapping. Each family thus groups together files in the file system that share at least some blocks among one another based on snapshot activities. Distinct files that do not share blocks based on snapping are provided in separate families. The file system leverages the snap-based relationships among family members to produce more accurate estimates of liability than would otherwise be feasible.

Managing data storage reservations on a per-family basis

Examples are generally directed towards a file server having a unified file system associated with a unified slice map volume. In one example, the unified file system stores file system metadata and sparse volume metadata within the unified slice map volume. The unified file system performs a unified set of functions using the file system metadata and sparse volume metadata in the unified slice map volume.

Unified slice map volume

A technique for storing support files in a data storage apparatus provides auxiliary files in a container file system. The container file system stores a container file that provides a file-based realization of a data object, such as an entire file system, LUN, or vVOL, for example, and stores one or more auxiliary files in the container file system alongside the container file. Thus, an auxiliary file that supports an application program's operations on a data object resides in the same fault domain with the container file that realizes the data object.

Auxiliary files in a container file system

A data storage system includes storage devices and a processing subsystem executing software forming a lower-deck file system and an upper-deck file system. The lower-deck file system presents a volume file from which storage is provided to the upper-deck file system, and units of storage of the storage devices are allocable to the upper-deck file system but are not reserved to it. The volume file is thinly provisioned, and additional units of storage are added dynamically to increase its allocated size in response to demand from the upper-deck file system. The lower-deck file system operates in a mapped mode in which a block address in the lower-deck file system is obtained by a generally arbitrary mapping (e.g., an Inode/IB tree) of a block address in the volume file.

File system over thinly provisioned volume file in mapped mode

A data storage system includes storage devices and a processing subsystem executing software forming a lower-deck file system and an upper-deck file system. The lower-deck file system presents a volume file from which storage is provided to the upper-deck file system, and units of storage of the storage devices are reserved to the upper-deck file system. The volume file is fully provisioned based on the reserved units of storage so that an out-of-space condition is not presented to the upper-deck file system. Reserved units of storage are dynamically added to the volume file to increase its allocated size in response to demand from the upper-deck file system, and a direct operating mode is used in which a block address in the lower-deck file system is obtained by a predetermined calculation from a corresponding block address in the volume file.

File system over fully provisioned volume file in direct mode

A data storage system employs a multi-level stack including a lower-deck file system and an upper-deck file system providing a file system interface to a file system consumer. The lower-deck file system presents a volume file to provide storage to the upper-deck file system. Storage is reserved to the upper-deck file system, and the volume file is fully provisioned so an out-of-space condition is not presented to the upper-deck file system. In an initial direct operating mode a block address in the lower-deck file system is obtained by a calculation from a corresponding block address in the volume file. Snapshots of a primary volume file are created and stored in respective snapshot volume files. The lower-deck file system transitions into a mapped mode using generally arbitrary mappings of block addresses of the primary volume file and snapshot volume files to obtain corresponding block addresses in the lower-deck file system.

Growing community of inventors

Fanwood, NJ, United States of America

Michal Marko