Hewlett Packard Enterprise Development LP

Simplivity Corporation

John Michael Czerkowicz

Glenn S Watkins

Sowmya Manjanatha

Arthur James Beaverson

Kishore Chitrapu

Peter Madany

Gareth Richards

Steven Bagby

Curtis Mello

Michael Champigny

Suparna Bhattacharya

Peter Thomas Camble

Syama Sundararao Nadiminti

Ieuan James Henry

Yan Xiao

Annmary Justine Koomthanam

Keshetti Mahesh

George A Klarakis

Sumit Narayan

Arthur J Beaverton

Glenn Watkins

Example implementations relate to partial file system instances. In an example, a subset of objects of a source file system instance on a source system are replicated to a target system to form a partial file system instance on the target system comprised of the subset of objects. Each of the objects of the source file system instance is identified by a signature based on content of each of the objects and the objects exhibit a hierarchical relationship to a root object in the file system instance. An unmaterialized object is dynamically added to the partial file system instance by replicating the corresponding object from the source file system instance. The target system is asynchronously updated from the source file system instance based on a comparison of the partial file system instance to the source file system instance.

Partial file system instances

Techniques and architectures to estimate storage system capacity are disclosed. Deduplicated data and an index of object entries are maintained in a storage system. The entries contain shallow reference counts. The shallow reference count values indicate a number of parent metadata objects that hold a reference to corresponding object. One or more tree milli-indices and one or more node milli-indices of object entries are maintained in the storage system. The entries correspond to a subset of objects stored in the storage system. The entries also have truncated object signature values and deep reference count values for the corresponding objects in the storage system. A capacity utilization of the storage system is determined based on analysis utilizing the deep reference count values to perform various multiset operations.

Storage system capacity usage estimation

Example implementations relate to encrypting data objects. In an example, data objects of a file system instance contained by a security domain are encrypted using a Data Encryption Key that is specific to the security domain and is wrapped by a Key Encryption Key shared exclusively within a cluster. A backup of the file system instance is created on a backup node. The backup includes at least some of the encrypted data objects. The DEK is sent to the backup node. The backup node cannot decrypt the backup unless the backup node is a member of the cluster and has access to the KEK to unwrap the DEK.

Backups of file system instances of encrypted data objects

In an example, a system may include an object store for storing objects, wherein each object is identified by an object signature generated according to a first cryptographic function, wherein the objects stored in the object store exhibit a hierarchical relationship from a root object. The system may receive a second cryptographic function for the object store, receive an I/O request affecting an object in the object store, encode the affected object according to the second cryptographic function, and persist the affected object alongside other objects in the object store encoded according to the first cryptographic function.

Object signatures in object stores

Example implementations relate to encrypting data objects. In an example, data objects of a file system instance contained by a security domain are encrypted using a Data Encryption Key that is specific to the security domain and is wrapped by a Key Encryption Key shared exclusively within a cluster. A root object of the file system instance is encrypted using a Metadata Encryption Key. A backup of the file system instance is created on a backup node. The Data Encryption Key and the Metadata Encryption Key are sent to the backup node.

Backups of file system instances with root object encrypted by metadata encryption key

In some examples, a method can involve generating data object records representing data objects in storage and storing, on a device, a lookup index including a plurality of index buckets, each index bucket being associated with a set of storage units on the device and including a respective index filter for each respective one of the set of storage units, the set of storage units containing a respective portion of the data object records, wherein each respective index filter includes at least one lookup entry for each data object record in the respective one of the set of storage units. The method can also involve receiving a data object request, and performing a lookup operation including identifying an index bucket for the data object and determining whether any of the set of storage units associated with the index bucket contains a data object record corresponding to the data object.

Optimized record lookups

In example implementations, a system may receive an input/output (I/O) operation in an object store storing a first object, determine objects represented in the object store affected by storing the first object, transcode the I/O operation into a transaction including the linked list of updates to objects represented in the object store, determine a persistent storage for storing the transaction and persist the transaction to the persistent storage according to a data structure determined by the type of persistent storage.

Transaction encoding and transaction persistence according to type of persistent storage

Example implementations relate to backups to a secondary storage system of a file system instance that hierarchically relates data objects by content-based signatures to a root object and includes subtrees that correspond to respective inodes. In an example, a list of the content-based signatures is sent to the secondary storage system. Metadata and data of a subtree is stored into an item in the secondary storage system designated for the subtree.

Backups of file system instances having subtrees

In an example, a system may modify an object stored in an object store for storing objects to include augmented metadata. The system may encode a signature of the modified object such that the signature is identical to a signature of the object prior to being modified to include the augmented metadata.

Augmented metadata and signatures for objects in object stores

In an example, a system includes network nodes implementing an object store. The system may determine reference count updates for objects stored on the object store, and delay persisting the reference count updates to a persistent storage. The system may cancel reference count updates that increments and decrement between the determined reference count update and another reference count update received during the delay in persisting the reference count update to the persistent storage to minimize writing to the object record.

Object counts persistence for object stores

Example implementations relate to allocating an I/O request. In an example, a demultiplexer may forward an I/O request to a file system instance to which the I/O request belongs. The file system instance may tag the I/O request with a file system instance identifier associated with that file system instance. A volume manager may identify an extent pool to which the I/O request is to be allocated from among a plurality of extent pools of a storage based on the file system instance identifier tagged to the I/O request.

Extent pool allocations based on file system instance identifiers

Example implementations relate to objects in extents on a storage resource. In an example, an extent identifier is persisted to a storage allocation table, where the extent identifier locates an extent on a storage resource. An extent map tracks which objects in the extent are deallocated. A virtual block address is generated allocating an object to store data. The virtual block address may include a storage allocation table key that locates the extent identifier in the storage allocation table and an extent offset to locate the object within the extent.

Virtual block addresses

Apparatus and method for placing data based on the content of the data in random access memory such that indexing operations are not required. A strong (e.g., cryptographic) hash is applied to a data element resulting in a signature. A weaker hash function is then applied to the signature to generate a storage location in memory for the data element. The weaker hash function assigns multiple data elements to the same storage location while the signature comprises a unique identifier for locating a particular data element at this location. In one embodiment a plurality of weak hash functions are applied successively to increase storage space utilization. In other embodiments, the assigned storage location can be determined by one or more attributes of the data element and/or the storage technology, e.g, long-lived versus short-lived data and/or different regions of the memory having different performance (e.g., access latency memory lifetime) characteristics.

Method and apparatus for content derived data placement in memory

Methods and systems are provided for tracking object instances stored on a plurality of network nodes, which tracking enables a global determination of when an object has no references across the networked nodes and can be safely de-allocated. According to one aspect of the invention, each node has a local object store for tracking and optionally storing objects on the node, and the local object stores collectively share the locally stored instances of the objects across the network. One or more applications, e.g., a file system and/or a storage system, use the local object stores for storing all persistent data of the application as objects.

Growing community of inventors

Wakefield, MA, United States of America

John Michael Czerkowicz