Datadirect Networks, Inc.

Other

Paul Joseph Nowoczynski

Jason Micah Cope

Michael Piszczek

Pavan Kumar Uppu

Donald Joseph Molaro

John Gordon Manning

Gordon Manning

Don Molaro

Michael Vildibill

Pavan Kumar Kumar Uppu

An improvement in erasure coded group management shared nothing storage clusters establishing an invariant which declares read-modify-writes will not be done by the system and implements a metadata distribution operation to provide fault tolerance.

Method for efficient erasure coded group management in shared nothing storage clusters

A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System's dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The 'write' request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata. The primary BB node gathers the metadata from the participating BB nodes, sends the metadata list to the BBNA unit, responsive to which the BBNA unit allocates a buffer sufficient to store the full data stripe, and transfers data fragments from participating BB nodes into the full data stripe buffer, thereby coalescing the data fragments into the full data stripe, which is subsequently transferred from the buffer in the BBNA unit into the File System.

Method for data transfer between compute clusters and file system

Data storage systems and methods for storing data in computing nodes of a super computer or compute cluster are described herein. The super computer storage may be coupled with a primary storage system. In addition to a CPU and memory, non-volatile memory is included with the computing nodes as local storage. The super computer includes a plurality of computing groups, each including a plurality of computing nodes. There is one burst buffer fabric per group and one input/output node per group. When data bursts occur, data may be stored by a first computing node on the local storage of a second computing node in the computing group through the burst buffer fabric without interrupting the CPU in the second computing node. Further, the local storage of other computing nodes may be used to store redundant copies of data from a first computing node to make the super computer data resilient.

Minimizing micro-interruptions in high-performance computing

Method and system for data transfer between compute clusters and file system

The present invention is directed to data migration, and particularly, Parity Group migration, between high performance data generating entities and data storage structure in which distributed NVM arrays are used as a single intermediate logical storage which requires a global registry/addressing capability that facilitates the storage and retrieval of the locality information (metadata) for any given fragment of unstructured data and where Parity Group Identifier and Parity Group Information (PGI) descriptors for the Parity Groups' members tracking, are created and distributed in the intermediate distributed NVM arrays as a part of the non-deterministic data addressing system to ensure coherency and fault tolerance for the data and the metadata. The PGI descriptors act as collection points for state describing the residency and replay status of members of the Parity Groups.

Method and system for data migration between high performance computing architectures and file system using distributed parity group information structures with non-deterministic data addressing

Systems and methods for reducing metadata in a write-anywhere storage system are disclosed herein. The system includes a plurality of clients coupled with a plurality of storage nodes, each storage node having a plurality of primary storage devices coupled thereto. A memory management unit including cache memory is included in the client. The memory management unit serves as a cache for data produced by the clients before the data is stored in the primary storage. The cache includes an extent cache, an extent index, a commit cache and a commit index. The movement of data and metadata is by an interval tree. Methods for reducing data in the interval tree increase data storage and data retrieval performance of the system.

Reducing metadata in a write-anywhere storage system

Method and system for reclamation of distributed dynamically generated erasure groups for data migration between high performance computing architectures and data storage using non-deterministic data addressing

Data storage systems and methods for storing data are described herein. The storage system may be integrated with or coupled with a compute cluster or super computer having multiple computing nodes. A plurality of nonvolatile memory units may be included with computing nodes, coupled with computing nodes or coupled with input/output nodes. The input/output nodes may be included with the compute cluster or super computer, or coupled thereto. The nonvolatile memory units store data items provided by the computing nodes, and the input/output nodes maintain where the data items are stored in the nonvolatile memory units via a hash table distributed among the input/output nodes. The use of a distributed hash table allows for quick access to data items stored in the nonvolatile memory units even as the computing nodes are writing large amounts of data to the storage system quickly in bursts.

Data storage architecture and system for high performance computing incorporating a distributed hash table and using a hash on metadata of data items to obtain storage locations

Data storage systems and methods for storing data are described herein. The storage system includes a first storage node is configured to issue a first delivery request to a first set of other storage nodes in the storage system, the first delivery request including a first at least one data operation for each of the first set of other storage nodes and issuing at least one other delivery request, while the first delivery request remains outstanding, the at least one other delivery request including a first commit request for each of the first set of other storage nodes. The first node causes the first at least one data operation to be made active within the storage system in response to receipt of a commit indicator along with a delivery acknowledgement regarding one of the at least one other delivery request.

Maintaining order and fault-tolerance in a distributed hash table system

Data storage systems and methods for storing data are described herein. The storage system includes at least two data storage nodes for storing portions of a distributed hash table and related data. After a first node attempts to complete a write request at a second node and is unable to complete the request, the first node ceases responses to interactions from other nodes. Once the first node's failure to respond has caused a sufficient number of nodes to cease responding, the nodes enter a service mode to resolve the live lock. While in live lock, the nodes determine the oldest, unfulfilled request using a system-wide logical timestamp associated with write requests. Once the oldest request is determined, a removal vote to remove the non-responsive node from the group is initiated and, if other nodes agree, the non-responsive node is removed from the group of nodes.

Handling failed transaction peers in a distributed hash table

Data storage architecuture and system for high performance computing hash on metadata in reference to storage request in nonvolatile memory (NVM) location

Growing community of inventors

Brooklyn, NY, United States of America

Paul Joseph Nowoczynski