Emc IP Holding Company LLC

Dell Products Inc.

Emc Corporation

Max Laier

Anton Rang

John Brazel

Justin Husted

Ronald Steinke

Evgeny Popovich

Alain Hamel

Suraj Raju

Suraj Brahma Raju

Jonathan Mark Walton

Ron Steinke

Mayank Rajawat

Edward Cande

Ryan Libby

Lachlan McIlroy

Chris Stacey

Attilio Rao

Cornelis Van Rij

Dhanashri Parasharam Patil

Simarjot Kaur

Hwanju Kim

The described technology is generally directed towards maintaining a heat score per file based on a file's access/activity level, and using the heat score to re-tier files from faster to slower storage and vice-versa. Files on a slower tier that have more activity are thereby moved ('up-tiered') to a faster access storage tier, while the files that have less activity on the faster access storage tier are moved (“down-tiered”) to a slower access storage tier. The heat score for a file is increased based on more input-output operation activity, and decayed over time. The decaying can be based on a relative importance value (e.g., business value) assigned to the file or a domain that includes the file. Ordering the files based on their respective heat scores allows only a fraction of the files to be considered for re-tiering before the re-tiering moves are performed.

Heat score-based tiering of data between different storage tiers of a file system

A method for direct addressing in virtual addressed systems includes obtaining, by a system including a processor and in response to receiving a request to access a data item stored by a storage system, address information for the data item from a data structure. The address information includes a first physical storage address of the storage system, a first generation number associated the first physical storage address in the data structure, and an address redirector. The method also includes accessing, by the system and in response to the first generation number being determined to be different from a second generation number associated with the first physical storage address, the data item at a second physical storage address of the storage system instead of the first physical storage address, the second physical storage address being determined based on the address redirector.

Hybrid physical/virtual data addressing with generation indicators

Data lock transitioning associated with writing of files can be performed and managed. In connection with a write operation to write a portion of a file to a node of a node network, an access management component (AMC) can analyze file-related information indicative of a status associated with the portion of the file, wherein the status can relate to whether the file portion is associated with a streaming write, whether a protocol indication indicates a file handle associated with the file has been closed, or whether an advisory lock has been released by a client device. The AMC can determine whether to transition from an exclusive data lock associated with the portion of the file to a shared data lock with respect to the portion of the file, based on a result of analyzing the file-related information. The AMC can fine tune determinations or predictions of data lock transitions.

Lock release management in a distributed system

Techniques for address mirroring of a file system journal are presented. A transactional file system update can correspond to a write operation to be performed at a first participant node of a data storage system. A journal transfer component can transfer the transactional file system update from an initiator node of the data storage system to a first journal associated with the first participant node and recovery information relating to the update to a second journal associated with a second participant node of the data storage system. The recovery information can be determined based on the update. To facilitate the update, a journal descriptor block can be generated and sent to the second participant node, such block comprising the recovery information, which can include transaction state, block addresses involved in the transaction, error correction-related information, or a mirror block address set for the block addresses where desired.

Address mirroring of a file system journal

Systems and methods facilitating fault tolerance for transaction mirroring are described herein. A method as described herein can include receiving a commit command for a data transaction from an initiator node of the system, wherein the data transaction is associated with a first failure domain, and wherein the commit command is directed to a primary participant node and a secondary participant node of the system; determining whether a response to the commit command has been received at the primary participant node from the secondary participant node in response to the receiving; and, in response to determining that the response to the commit command was not received at the primary participant node, indicating that the secondary participant node is invalid in a data store associated with a second failure domain that is distinct from the first failure domain.

Fault tolerance for transaction mirroring

Systems and methods for mirroring a file system journal are described herein. A method as described herein can include receiving, by a system operatively coupled to a processor, transactional file system updates corresponding to a write operation to be performed at a first node of the system; transferring, by the system, the transactional file system updates from an initiator node of the system to a first journal at the first node and a second journal at a second node of the system that is logically distinct from the first node and the initiator node; and committing, by the system, the transactional file system updates to the first journal and the second journal.

System and method for mirroring a file system journal

Direct read in clustered file systems is described herein. A method as described herein can include determining, for a write operation on a resource stored by a data storage system, as initiated by an initiator node, a reference count for the resource, the reference count comprising a number of target storage regions of the data storage system to be modified by write data during the write operation; facilitating conveying, from the initiator node to a lock coordinator node, the reference count for the resource; facilitating conveying, from the initiator node to respective participant nodes that are respectively assigned to the target storage regions, the write data and a key value for the write operation; and facilitating causing the respective participant nodes to convey respective notifications that comprise the key value in response to the respective participant nodes writing the write data to the target storage regions.

Direct read in clustered file systems

Pre-decompressing a compressed form of data that has been pre-fetched into a cache to facilitate subsequent retrieval of a decompressed form of the data from the cache is presented herein. A system retrieves, from a first portion of a cache, a compression chunk comprising compressed data blocks representing a compressed form of a group of data blocks in response to a first cache hit from the first portion of the cache being incurred, decompresses the compression chunk to obtain a decompressed chunk comprising uncompressed data blocks representing an uncompressed form of the group of data blocks, and inserts the uncompressed data blocks into a second portion of the cache. Further, the system retrieves, from the second portion of the cache, an uncompressed data block of the uncompressed data blocks in response to a second cache hit from the second portion of the cache being incurred.

Pre-decompressing a compressed form of data that has been pre-fetched into a cache to facilitate subsequent retrieval of a decompressed form of the data from the cache

Compression of data for a file system utilizing protection groups can be implemented and managed. A compression management component (CMC) can control compression of data via inline or post-process compression for storage in protection groups in memory, including determining whether to compress data, determining a compression algorithm to utilize to compress data, and/or determining whether to perform inline and/or post-process compression of data. CMC can generate protection group (PG) metadata for a PG in which compressed data is stored. PG metadata can comprise a logical extent map that describes which logical blocks contain compressed data, a list of cyclic redundancy check values for logical blocks, and a list of compression chunks that store individual metadata regarding individual compressed streams, wherein, for an individual compressed stream, the individual metadata comprises a compression format, compressed size, uncompressed size, and/or starting offset in physical space within the PG.

Compression of data for a file system

Techniques are provided for computer memory mapping and allocation. In an example, a virtual memory address space is divided into an active half and a passive half. Processors make memory allocations to their respective portions of the active half until one processor has made a determined number of allocations. When that occurs, and when all memory in the passive half that has been allocated has been returned, then the active and passive halves are switched, and all processors are switched to making allocations in the newly-active half.

Computer memory mapping and invalidation

Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods as a part of tiering data to external repositories. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.

Persistent cache layer to tier data to cloud storage

Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods or store unmodified raw data in local storage. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.

Persistent cache layer in a distributed file system

Implementations are provided for efficient flushing of a transaction journal. A transaction journal can be maintained in memory of a node that stores writes to the targeted to the drives housed within the node. Drives within the node can be score for busyness when determining which transactions to flush from memory to disk. Journal entries relating to writes that have a high locality can be selected for flushing. Additionally, each block on each disk in a node can be tracked for past writes which can be used as active intelligence to predict when a journal entry may be rewritten in the future. Journal entries relating to blocks with a high probability to be overwritten can avoid being flushed. By increasing the efficiency of transaction journal flushing, a node can increase the throughput of data, have less busy disks, incur less write amplification, and support more coalescing.

Efficient transaction log flushing

Various embodiments are directed towards enabling data writes utilizing a node cache and a logstore stored on a stable storage device. A client device may send data to a node for writing to a parent file. The node may cache the received data prior to writing the data to the parent file. Caching the received data may comprise adding the received data to a node cache and to a logstore. In one embodiment, the node cache may include a coalescer that combines data from a plurality of data writes from the client device to the node prior to writing to the parent file. In some embodiments, the logstore may be mirrored logstore on one or more stable storage devices. The parent file may periodically be updated from the node cache data. After the parent file is updated, the node cache data and the logstore data may be purged.

Growing community of inventors

Seattle, WA, United States of America

Max Laier