Netapp, Inc.

Venkateswarlu Tella

Kartik Rathnakar

Cheryl Marie Thompson

Qinghua Zheng

Alok Sharma

Mukul Sharma

Palak Sharma

Girish Hebbale Venkata Subbaiah

Dnyaneshwar Nagorao Pawar

Dibyasri Nandi

Debanjan Paul

Dinakaran Narayanan

Sindhushree K N

Girish Hebbale Venkatasubbaiah

Ananthan Subramanian

Kiran Nenmeli Srinivasan

Rajesh Khandelwal

Rahul Thapliyal

Harsh Tiwari

Mathankumar Devarajan

Techniques are provided for mirroring objects between object stores hosted by cloud providers that could have different data layout requirements. An object may be stored within an object store that supports a unified object format where the object is capable of storing compressed data. The object may be mirrored to a destination object store that may also support the unified object format or to a destination object store that does not support the unified object format. If the destination object store does not support the unified object format, then slot header metadata within the object is used to decompress the data within the object into an uncompressed format. The data is packaged from being in the uncompressed format into a fixed offset format supported by the destination object store to create a mirrored object that is stored into the destination object store while retaining compression of the data.

Mirroring objects between different cloud providers

Techniques are provided for mirroring objects between object stores hosted by cloud providers that have different data layout requirements. An object may be stored within a first object store that supports a fix offset format where uncompressed data is stored according to fixed offsets and boundaries within fixed size objects. A mirroring operation may be used to mirror the object to a second object store that supports a unified object format where compressed data can be stored at non-fixed offsets and boundaries within variable sized objects. The mirroring operation selects a compression algorithm and compresses the object on the fly to create a mirrored object having the unified object format. The mirrored object, populated with the compressed data and slot header metadata comprising compression information for how to locate and decompress the data in the mirrored object, is stored into the second object store.

Mirroring objects between different cloud providers with different data layout requirements

Techniques are provided for implementing a unified object format. The unified object format is used to format data in a performance tier (e.g., infrequently accessed data, snapshot data, etc.) into objects that are stored into an object store for low cost, scalable, long term storage compared to storage of the performance tier. With the unified object format, compression of the data may be retained when the data is stored as the objects into the object store. Additional compression may also be provided for the data in the objects. The unified object format includes slot header metadata used to track the location of the data within the object notwithstanding the data being compressed and/or stored at non-fixed boundaries. The slot header metadata may be cached at the performance tier for improved read performance and may be repaired by a repair subsystem (a slot header repair subsystem).

Persistent metafile used to cache slot header metadata for improved read performance of objects within an object store

Repair subsystem for persistent metafile repair to improve resiliency

Unified object format for retaining compression and performing additional compression for reduced storage consumption in an object store

Techniques are provided for a layout format for compressed data. A first set of data blocks are grouped into a first group based upon a first frequency of access to the first set of data blocks. A second set of data blocks are grouped into a second group based upon a second frequency of access to the second set of data blocks. The first set of data blocks are compressed into a first compression group using a first compression algorithm. The second set of data blocks are compressed into a second compression group using a second compression algorithm.

Layout format for compressed data

Methods and systems for a storage environment are provided. One method includes identifying, by a processor, a plurality of block numbers of a fragmented address space for re-allocation, each block number associated with data stored by a file system in a storage device of a storage system; determining, by the processor, compressed data associated with a block number from among the plurality of block numbers; verifying, by the processor, that an indirect block of a hierarchical structure maintained by the file system references the block number associated with the compressed data; and copying, by the processor, the compressed data to a new block, without decompressing the data.

Block allocation methods and systems in a networked storage environment

Techniques are provided for asynchronous semi-inline deduplication. A multi-tiered storage arrangement comprises a first storage tier, a second storage tier, etc. An in-memory change log of data recently written to the first storage tier is evaluate to identify a fingerprint of a data block recently written to the first storage tier. A donor data store, comprising fingerprints of data blocks already stored within the first storage tier, is queried using the fingerprint. If the fingerprint is found, then deduplication is performed for the data block to create deduplicated data based upon a potential donor data block within the first storage tier. The deduplicated data is moved from the first storage tier to the second storage tier, such as in response to a determination that the deduplicated data has not been recently accessed. The deduplication is performed before cold data is moved from first storage tier to second storage tier.

Asynchronous semi-inline deduplication

One or more techniques and/or computing devices are provided for inline deduplication. For example, a checksum hash table and/or a block number hash table may be maintained within memory (e.g., a storage controller may maintain the hash tables in-core). The checksum hash table may be utilized for inline deduplication to identify potential donor blocks that may comprise the same data as an incoming storage operation. Data within an in-core buffer cache is eligible as potential donor blocks so that inline deduplication may be performed using data from the in-core buffer cache, which may mitigate disk access to underlying storage for which the in-core buffer cache is used for caching. The block number hash table may be used for updating or removing entries from the hash tables, such as for blocks that are no longer eligible as potential donor blocks (e.g., deleted blocks, blocks evicted from the in-core buffer cache, etc.).

Inline deduplication

Growing community of inventors

Bengaluru, India

Venkateswarlu Tella