Hewlett Packard Enterprise Development LP

Glenn S Watkins

John Michael Czerkowicz

Peter Madany

Gareth Richards

Curtis Mello

Michael Champigny

Yan Xiao

George A Klarakis

Sumit Narayan

Techniques and architectures to estimate storage system capacity are disclosed. Deduplicated data and an index of object entries are maintained in a storage system. The entries contain shallow reference counts. The shallow reference count values indicate a number of parent metadata objects that hold a reference to corresponding object. One or more tree milli-indices and one or more node milli-indices of object entries are maintained in the storage system. The entries correspond to a subset of objects stored in the storage system. The entries also have truncated object signature values and deep reference count values for the corresponding objects in the storage system. A capacity utilization of the storage system is determined based on analysis utilizing the deep reference count values to perform various multiset operations.

Storage system capacity usage estimation

Implementations of the disclosure describe an expandable index including pages that may store object records. The index may be divided into buckets, and each bucket may be associated with pages from a pool of unused pages shared by the buckets. A table or log may be used to maintain a dynamic list of all unused pages. It may also maintain a list of pages currently associated with buckets. As pages are linked to or unlinked from a bucket, the table or log may be updated to reflect that a page has been linked to a bucket or that a page that is no longer linked to a bucket belongs to the pool of unused pages. As additional storage media are added to the system, additional pages may be added to the pool of the expandable index for use by existing buckets without changing the number of buckets of the index.

Expandable index with pages to store object records

Example implementations relate to encrypting data objects. In an example, data objects of a file system instance contained by a security domain are encrypted using a Data Encryption Key that is specific to the security domain and is wrapped by a Key Encryption Key shared exclusively within a cluster. A backup of the file system instance is created on a backup node. The backup includes at least some of the encrypted data objects. The DEK is sent to the backup node. The backup node cannot decrypt the backup unless the backup node is a member of the cluster and has access to the KEK to unwrap the DEK.

Backups of file system instances of encrypted data objects

In an example, a system may include an object store for storing objects, wherein each object is identified by an object signature generated according to a first cryptographic function, wherein the objects stored in the object store exhibit a hierarchical relationship from a root object. The system may receive a second cryptographic function for the object store, receive an I/O request affecting an object in the object store, encode the affected object according to the second cryptographic function, and persist the affected object alongside other objects in the object store encoded according to the first cryptographic function.

Object signatures in object stores

A method for storing data in a storage system, includes opportunistically compressing a plurality of objects of a first size during a write operation, storing the compressed objects of the first size if they compress acceptably, and storing other objects of a second size uncompressed. It further includes determining during a read operation whether an object of the second size is stored as a part of a compressed object; if the object of the second size is not stored as a part of a compressed object of the first size, then reading the object of the second size from storage; if the object of the second size is stored as a part of the compressed object of the second size, then: reading the compressed object of the first size from storage; uncompressing the compressed object of the first size; and extracting the object of the second size.

Opportunistic compression

Example implementations relate to encrypting data objects. In an example, data objects of a file system instance contained by a security domain are encrypted using a Data Encryption Key that is specific to the security domain and is wrapped by a Key Encryption Key shared exclusively within a cluster. A root object of the file system instance is encrypted using a Metadata Encryption Key. A backup of the file system instance is created on a backup node. The Data Encryption Key and the Metadata Encryption Key are sent to the backup node.

Backups of file system instances with root object encrypted by metadata encryption key

In some examples, a method can involve generating data object records representing data objects in storage and storing, on a device, a lookup index including a plurality of index buckets, each index bucket being associated with a set of storage units on the device and including a respective index filter for each respective one of the set of storage units, the set of storage units containing a respective portion of the data object records, wherein each respective index filter includes at least one lookup entry for each data object record in the respective one of the set of storage units. The method can also involve receiving a data object request, and performing a lookup operation including identifying an index bucket for the data object and determining whether any of the set of storage units associated with the index bucket contains a data object record corresponding to the data object.

Optimized record lookups

In example implementations, a system may receive an input/output (I/O) operation in an object store storing a first object, determine objects represented in the object store affected by storing the first object, transcode the I/O operation into a transaction including the linked list of updates to objects represented in the object store, determine a persistent storage for storing the transaction and persist the transaction to the persistent storage according to a data structure determined by the type of persistent storage.

Transaction encoding and transaction persistence according to type of persistent storage

In an example, a system may modify an object stored in an object store for storing objects to include augmented metadata. The system may encode a signature of the modified object such that the signature is identical to a signature of the object prior to being modified to include the augmented metadata.

Augmented metadata and signatures for objects in object stores

In an example, a system includes network nodes implementing an object store. The system may determine reference count updates for objects stored on the object store, and delay persisting the reference count updates to a persistent storage. The system may cancel reference count updates that increments and decrement between the determined reference count update and another reference count update received during the delay in persisting the reference count update to the persistent storage to minimize writing to the object record.

Object counts persistence for object stores

Example implementations relate to allocating an I/O request. In an example, a demultiplexer may forward an I/O request to a file system instance to which the I/O request belongs. The file system instance may tag the I/O request with a file system instance identifier associated with that file system instance. A volume manager may identify an extent pool to which the I/O request is to be allocated from among a plurality of extent pools of a storage based on the file system instance identifier tagged to the I/O request.

Extent pool allocations based on file system instance identifiers

Example implementations relate to objects in extents on a storage resource. In an example, an extent identifier is persisted to a storage allocation table, where the extent identifier locates an extent on a storage resource. An extent map tracks which objects in the extent are deallocated. A virtual block address is generated allocating an object to store data. The virtual block address may include a storage allocation table key that locates the extent identifier in the storage allocation table and an extent offset to locate the object within the extent.

Growing community of inventors

Westborough, MA, United States of America

Glenn S Watkins