Netapp, Inc.

Network Appliance, Inc.

Sandeep Yadav

Subramaniam Periyagaram

Blake H Lewis

Ashish Prakash

Rickard Edward Faith

Ananthan Subramanian

Dnyaneshwar Nagorao Pawar

Kevin Daniel Varghese

Parag Sarfare

Rajesh Khandelwal

Suhas Urkude

John K Edwards

Subramanian Periyagaram

David A Grunwald

Ram Kesavan

Konstantinos Roussos

Don Alvin Trimmer

Rahul Khona

Yasuhiro Endo

Jim Voll

Jeffrey S Kimmel

Stephen Wu

Yong Cho

Varun Jobanputra

Sriram S Rao

Stephen William Daniel

John Zhuge

Anureita Rao

Venkatesh Babu Ks

David Crunwald

Subramaniam Perivagaram

Anand Natarajan

Pratap Singh

Jeff Kimmel

Steve Daniel

Methods and systems for co-locating journaling and data storage are provided. Separate journal and volume partitions may be maintained within each logical storage unit (e.g., Logical Unit Number (LUN)) of a distributed storage system. Journaling of metadata associated with write requests received from one or more clients may be distributed by identifying a destination logical storage unit to which data associated with a given write request is to be stored and causing the data and metadata to be persisted to disk by journaling the metadata and the data to respective portions of an active log within the journal partition of the destination logical storage unit. By using the same logical storage unit for both journaling of write requests and writing the data associated with such write requests, the bottleneck due to there being only a single device or storage unit handling all metadata for all write requests can be avoided.

Co-located journaling and data storage for write requests

A method and system for co-locating journaling and data storage based on write requests. A write request that includes metadata and data is received from a client. A logical storage unit for storing the metadata and the data is identified. The logical storage unit is divided into a journal partition and a volume partition. The journal partition includes a first log and a second log. Which of the first log and the second log is an active log and which of the first log and the second log is an inactive log are identified. The metadata is recorded in a first location in the active log and the data is recorded in a second location in the active log during a single I/O operation. A reply is sent to the client after the metadata and the data are recorded in the journal partition.

Partially overwriting a compression group without decompressing compressed data can consumption of resources for the decompression. A storage server partially overwrites the compression group when a file block identifier of a client's write request resolves to the compression group. The compression group remains compressed while the partial overwriting is performed.

Overwriting part of compressed data without decompressing on-disk compressed data

A network adapter receives a request to store a data block. The data block is sent from the network adapter to a compression module. The compression module generates a compressed data block from the data block. The compressed data block or a reference to the compressed data block is stored in a buffer cache. The compressed data block is stored in nonvolatile memory. It is determined that the compressed data block should be flushed a storage device. In response to determining that the compressed data block should be flushed to the storage device, the compressed data block is flushed from the nonvolatile memory to the storage device.

Nearstore compression of data in a storage system

It is determined that a first data block contains the same data as a second data block. The first data block is associated with a first extent and the second data block is associated with a second extent. In response to determining that the first data block contains the same data as the second data block, the second data block is associated with the first extent and the first data block is disassociated with the second extent.

Deduplication in an extent-based architecture

A method and apparatus for retention of active data stored in memory using multiple indexing systems for data storage. An embodiment of a method for retention of active data in a storage server includes reading data into a first location of a main memory of the storage server. The data in the first location indexes data elements in a long-term data storage in a first manner. The method further provides for copying the data from the first location into a second location in the main memory of the storage server, where the data in the second location indexing the data elements in the long-term data storage in a second manner.

Retention of active data stored in memory

Provided is a method and system for reducing duplicate buffers in buffer cache associated with a storage device. Reducing buffer duplication in a buffer cache includes accessing a file reference pointer associated with a file in a deduplicated filesystem when attempting to load a requested data block from the file into the buffer cache. To determine if the requested data block is already in the buffer cache, aspects of the invention compare a fingerprint that identifies the requested data block against one or more fingerprints identifying a corresponding one or more sharable data blocks in the buffer cache. A match between the fingerprint of the requested data block and the fingerprint from a sharable data block in the buffer cache indicates that the requested data block is already loaded in buffer cache. The sharable data block in buffer cache is used instead thereby reducing buffer duplication in the buffer cache.

Single instance buffer cache method and system

An extent-based storage architecture is implemented by a storage server receiving a read request for an extent from a client, wherein the extent includes a group of contiguous blocks and the read request includes a file block number. The storage server retrieves an extent identifier from a first sorted data structure, wherein the storage server uses the received file block number to traverse the first sorted data structure to the extent identifier. The storage server retrieves a reference to the extent from a second sorted data structure, wherein the storage server uses the retrieved extent identifier to traverse the second sorted data structure to the reference, and wherein the second sorted data structure is global across a plurality of volumes. The storage server retrieves the extent from a storage device using the reference and returns the extent to the client.

Extent-based storage architecture

An extent-based storage architecture is implemented by a storage server. The storage server generates a new extent identifier for cloning a source extent identified by a source extent identifier and stored at a source data structure that includes a length value providing a length of the source extent, an offset value and a reference count value that provides a number of data containers that reference the source extent identifier. The storage server updates a data structure for a cloned version of the data container for storing the new extent identifier that points to the source extent identifier and includes an extent length value and offset value different from length value and the offset value of the source data structure.

Cloning using an extent-based architecture

A system and method for transparently compressing file system data using compression group descriptors is provided. When data contained within a compression group be compressed beyond a predefined threshold value, a compression group descriptor is included in the compression group that signifies that the data for the group of level 0 blocks is compressed into a lesser number of physical data blocks. When performing a read operation, the file system first determines the appropriate compression group that contains the desired data and determines whether the compression group has been compressed. If so, the file system decompresses the data in the compression group before returning the decompressed data. If the magic value is not the first pointer position, then the data within the compression group was previously stored in an uncompressed format, and the data may be returned without performing a decompression operation.

System and method for file system level compression using compression group descriptors

A method and system for eliminating the redundant allocation and deallocation of special data on disk, wherein the redundant allocation and deallocation of special data on disk is eliminated by providing an innovate technique for specially allocating special data of a storage system. Specially allocated data is data that is pre-allocated on disk and stored in memory of the storage system. 'Special data' may include any pre-decided data, one or more portions of data that exceed a pre-defined sharing threshold, and/or one or more portions of data that have been identified by a user as special. For example, in some embodiments, a zero-filled data block is specially allocated by a storage system. As another example, in some embodiments, a data block whose contents correspond to a particular type document header is specially allocated.

Use of predefined block pointers to reduce duplicate storage of certain data in a storage subsystem of a storage server

Method and system for partially cloning a data container with compression is provided. A storage operating system determines if a portion of a source data container that is to be cloned includes a plurality of compressed blocks that are compressed using a non-variable compression group size. The operating system clones the plurality compressed blocks with the non-variable compression group size and de-compresses a plurality of blocks of the data container that are not within the non-variable compression group size. The plurality of compressed blocks and the plurality of blocks that are not within the non-variable compression group size are then stored as a partially cloned copy of the source data container.

Method and system for using non-variable compression group size in partial cloning

A method for efficiently handling partial write requests in a storage system includes allocating a new block of data for the new partial data, and allocating a record in an extent map to record the location of the new partial data block, the location of the old partial data block and the offset length for each data block. Data blocks can be repackaged in the background when system resources are available. A full, but misaligned write request is also efficiently handled by writing the new data to a newly allocated data block and allocating new records in an extent map to record information corresponding to two partial write operations.

Growing community of inventors

Santa Clara, CA, United States of America

Sandeep Yadav