Datadirect Networks, Inc.

Jason Micah Cope

Michael Piszczek

Pavan Kumar Uppu

Paul Joseph Nowoczynski

William Joseph Harker

Thomas E Fugini

Donald Joseph Molaro

John Gordon Manning

Gordon Manning

Zhiwei Sun

Don Molaro

Eric Barton

Yuhua Guo

Michael Vildibill

Pavan Kumar Kumar Uppu

A system and method for efficiently storing and accessing large volumes of metadata persistent on Non-Volatile Memory (NVM) storage systems is provided. The system applies log-structured, Copy-on-Write (CoW) Btree methods, and supports a core-affine data and resource partitioning approaches on the system's architecture and platform with a high-degree of parallelism within the CPU, NVMe storage, and networking devices. The subject system and method efficiently indexes both in-core (DRAM resident) and out-of-core (NVM resident) metadata, supports a variety of data access patterns, supports CoW features and provides verifiable data safety and integrity capabilities. The present system minimizes latencies over all aspects of the metadata management and access path by leveraging core-affine resource partitioning with runtime environment providing lightweight user-level threads with low-latency context switching that execute within the exclusive context of a dedicated CPU core, and partitioned resources.

System and method for non-volatile memory-based optimized, versioned, log-structured metadata storage with efficient data retrieval

System and method for non-volatile memory based optimized, versioned, log-structured metadata storage with efficient data retrieval

A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System's dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The 'write' request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata. The primary BB node gathers the metadata from the participating BB nodes, sends the metadata list to the BBNA unit, responsive to which the BBNA unit allocates a buffer sufficient to store the full data stripe, and transfers data fragments from participating BB nodes into the full data stripe buffer, thereby coalescing the data fragments into the full data stripe, which is subsequently transferred from the buffer in the BBNA unit into the File System.

Method for data transfer between compute clusters and file system

A data migration system supports a low-latency and reduced overhead data storage protocol for data storage sharing in a non-collision fashion which does not require inter-communication and permanent arbitration between data storage controllers to decide on the data placement/routing. The multiple data fragments of data sets are prevented from routing to the same storage devices by a multi-step selection protocol which selects (in a first phase of the selection routine) a healthy highest ranked drive enclosure, and further selects (in a second phase of the selection routine) a healthy highest-ranked data storage controller residing in the selected drive enclosure, for routing data fragments to different storage pools assigned to the selected data storage devices for exclusive 'writing' and data modification. The selection protocol also contemplates various failure scenarios in a data placement collision free manner.

Low latency and reduced overhead data storage system and method for sharing multiple storage devices by high performance computing architectures

The system and routine for data caching leverages the properties of Network-Attached Non-Volatile Memories (NANVMs) to provide virtualized secure node-local storage services to the network users with reduced data movement across the NANVMs. The caching routine reserves storage resources (storage partitions) on NANVM devices, migrates data required for the target application execution to the allocated storage partitions, and directs the network clients to dynamically 'mount' to the storage partitions based on application data requirements. Only those clients and applications that present valid credentials and satisfactory computing capabilities can access the data in the specific storage partitions. Several clients can have an access to the same storage partitions without duplication or replicating the data. A Global Data Indexing sub-system supports the efficient operation of the subject system. The Global Data Indexing Sub-System provides mapping between the storage partitions, data sets, applications, client nodes, as well as their credentials/capabilities.

System and method for scale-out node-local data caching using network-attached non-volatile memories

Variable Redundancy Distributed (VRD) RAID controller in a data storage environment contains embedded RAID logic permitting to choose and compute a desired redundancy coding scheme from a plurality thereof pre-programmed and embedded in a Compute Engine in the VRD RAID controller. 'Write' or 'Read' requests which are received from data generating entities, contain information identifying a type of the redundancy coding scheme of interest. The controller decodes the request, and automatically applies the desired computation to the incoming data without burdening the CPU with the computational activity. The variable redundancy computational ability of the subject systems provides an extremely versatile and flexible tool for RAID operations.

Data storage system employing a variable redundancy distributed RAID controller with embedded RAID logic and method for data migration between high-performance computing architectures and data storage devices using the same

Data storage systems and methods for storing data in computing nodes of a super computer or compute cluster are described herein. The super computer storage may be coupled with a primary storage system. In addition to a CPU and memory, non-volatile memory is included with the computing nodes as local storage. The super computer includes a plurality of computing groups, each including a plurality of computing nodes. There is one burst buffer fabric per group and one input/output node per group. When data bursts occur, data may be stored by a first computing node on the local storage of a second computing node in the computing group through the burst buffer fabric without interrupting the CPU in the second computing node. Further, the local storage of other computing nodes may be used to store redundant copies of data from a first computing node to make the super computer data resilient.

Minimizing micro-interruptions in high-performance computing

Method and system for data migration between data generating entities and data storage devices protected by de-clustered RAID algorithm are enhanced by dynamically controlling the I/O activity towards the data storage devices (NVM devices) based on their remaining lifespan (health) with the goal to prevent multiple devices selected for writing a parity stripe information from simultaneous failures. This feature is rendered by polling the remaining health of NVM devices in the RAID pool, computing a weighted lifespan for each NVM device, comparing the latter to an average of all NVM devices in the pool, and adjusting the I/O activity towards the NVM device of interest accordingly. If the weighted lifespan exceeds the average lifespan in the pool, the allowed I/O activity is increased, and if the weighted lifespan is below the average for the pool, then the device in question is sent less 'writes'.

Method and system for data migration between high performance computing entities and a data storage supported by a de-clustered raid (DCR) architecture with I/O activity dynamically controlled based on remaining health of data storage devices

In the data storage system the storage area network performs XOR operations on incoming data for parity generation without buffering data through a centralized RAID engine or processor. The hardware for calculating the XOR data is distributed to incrementally calculate data parity in parallel across each data channel and may be implemented as a set of FPGAs with low bandwidths to efficiently scale as the amount of storage memory increases. A host adaptively appoints data storage controllers in the storage area network to perform XOR parity operations on data passing therethrough. The system provides data migration and parity generation in a simple and effective matter and attains a reduction in cost and power consumption.

Data storage system and method for data migration between high-performance computing architectures and data storage devices using storage controller with distributed XOR capability

Method and system for data transfer between compute clusters and file system

A data storage system is implemented with an active power monitoring and control performed by a control node elected among a number of nodes. A real-time power monitoring information is supplied to the control node from, a power monitoring logic residing at each device in the system. The devices in the data storage system are pre-allocated with respective individual power budgets which are below the maximum power usage thereof. The power budgets of all the equipment cumulatively constitute a power budget assigned to the group of equipment. The control node controls dynamically and in real time power sharing between the plurality devices so that the devices with required power usage below the pre-allocated power budget can share their extra power credits with devices which are in need for extra power for performing its operation. The control node provides sharing of the power among the equipment in the data storage system with a goal of avoiding exceeding of the cumulative power budget assigned for the entire system or the cluster of the equipment.

Data storage system with active power management and method for monitoring and dynamical control of power sharing between devices in data storage system

The present invention is directed to data migration, and particularly, Parity Group migration, between high performance data generating entities and data storage structure in which distributed NVM arrays are used as a single intermediate logical storage which requires a global registry/addressing capability that facilitates the storage and retrieval of the locality information (metadata) for any given fragment of unstructured data and where Parity Group Identifier and Parity Group Information (PGI) descriptors for the Parity Groups' members tracking, are created and distributed in the intermediate distributed NVM arrays as a part of the non-deterministic data addressing system to ensure coherency and fault tolerance for the data and the metadata. The PGI descriptors act as collection points for state describing the residency and replay status of members of the Parity Groups.

Method and system for data migration between high performance computing architectures and file system using distributed parity group information structures with non-deterministic data addressing

Systems and methods for reducing metadata in a write-anywhere storage system are disclosed herein. The system includes a plurality of clients coupled with a plurality of storage nodes, each storage node having a plurality of primary storage devices coupled thereto. A memory management unit including cache memory is included in the client. The memory management unit serves as a cache for data produced by the clients before the data is stored in the primary storage. The cache includes an extent cache, an extent index, a commit cache and a commit index. The movement of data and metadata is by an interval tree. Methods for reducing data in the interval tree increase data storage and data retrieval performance of the system.

Reducing metadata in a write-anywhere storage system

Method and system for reclamation of distributed dynamically generated erasure groups for data migration between high performance computing architectures and data storage using non-deterministic data addressing

Data storage systems and methods for storing data are described herein. The storage system may be integrated with or coupled with a compute cluster or super computer having multiple computing nodes. A plurality of nonvolatile memory units may be included with computing nodes, coupled with computing nodes or coupled with input/output nodes. The input/output nodes may be included with the compute cluster or super computer, or coupled thereto. The nonvolatile memory units store data items provided by the computing nodes, and the input/output nodes maintain where the data items are stored in the nonvolatile memory units via a hash table distributed among the input/output nodes. The use of a distributed hash table allows for quick access to data items stored in the nonvolatile memory units even as the computing nodes are writing large amounts of data to the storage system quickly in bursts.

Data storage architecture and system for high performance computing incorporating a distributed hash table and using a hash on metadata of data items to obtain storage locations

Growing community of inventors

Highland, MD, United States of America

Jason Micah Cope