Datadirect Networks, Inc.

Pavan Kumar Uppu

Michael Piszczek

Jason Micah Cope

Paul Joseph Nowoczynski

William Joseph Harker

Thomas E Fugini

John Gordon Manning

Gordon Manning

Don Molaro

Donald Joseph Molaro

Cedric T Fernandes

Manjari Mishra

Shivkumar Pandit

Dave Fellinger

Lee Douglas McBryde

Michael Vildibill

John Albert De Leon

A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System's dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The 'write' request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata. The primary BB node gathers the metadata from the participating BB nodes, sends the metadata list to the BBNA unit, responsive to which the BBNA unit allocates a buffer sufficient to store the full data stripe, and transfers data fragments from participating BB nodes into the full data stripe buffer, thereby coalescing the data fragments into the full data stripe, which is subsequently transferred from the buffer in the BBNA unit into the File System.

Method for data transfer between compute clusters and file system

The system and routine for data caching leverages the properties of Network-Attached Non-Volatile Memories (NANVMs) to provide virtualized secure node-local storage services to the network users with reduced data movement across the NANVMs. The caching routine reserves storage resources (storage partitions) on NANVM devices, migrates data required for the target application execution to the allocated storage partitions, and directs the network clients to dynamically 'mount' to the storage partitions based on application data requirements. Only those clients and applications that present valid credentials and satisfactory computing capabilities can access the data in the specific storage partitions. Several clients can have an access to the same storage partitions without duplication or replicating the data. A Global Data Indexing sub-system supports the efficient operation of the subject system. The Global Data Indexing Sub-System provides mapping between the storage partitions, data sets, applications, client nodes, as well as their credentials/capabilities.

System and method for scale-out node-local data caching using network-attached non-volatile memories

Variable Redundancy Distributed (VRD) RAID controller in a data storage environment contains embedded RAID logic permitting to choose and compute a desired redundancy coding scheme from a plurality thereof pre-programmed and embedded in a Compute Engine in the VRD RAID controller. 'Write' or 'Read' requests which are received from data generating entities, contain information identifying a type of the redundancy coding scheme of interest. The controller decodes the request, and automatically applies the desired computation to the incoming data without burdening the CPU with the computational activity. The variable redundancy computational ability of the subject systems provides an extremely versatile and flexible tool for RAID operations.

Data storage system employing a variable redundancy distributed RAID controller with embedded RAID logic and method for data migration between high-performance computing architectures and data storage devices using the same

Data storage systems and methods for storing data in computing nodes of a super computer or compute cluster are described herein. The super computer storage may be coupled with a primary storage system. In addition to a CPU and memory, non-volatile memory is included with the computing nodes as local storage. The super computer includes a plurality of computing groups, each including a plurality of computing nodes. There is one burst buffer fabric per group and one input/output node per group. When data bursts occur, data may be stored by a first computing node on the local storage of a second computing node in the computing group through the burst buffer fabric without interrupting the CPU in the second computing node. Further, the local storage of other computing nodes may be used to store redundant copies of data from a first computing node to make the super computer data resilient.

Minimizing micro-interruptions in high-performance computing

In the data storage system the storage area network performs XOR operations on incoming data for parity generation without buffering data through a centralized RAID engine or processor. The hardware for calculating the XOR data is distributed to incrementally calculate data parity in parallel across each data channel and may be implemented as a set of FPGAs with low bandwidths to efficiently scale as the amount of storage memory increases. A host adaptively appoints data storage controllers in the storage area network to perform XOR parity operations on data passing therethrough. The system provides data migration and parity generation in a simple and effective matter and attains a reduction in cost and power consumption.

Data storage system and method for data migration between high-performance computing architectures and data storage devices using storage controller with distributed XOR capability

Method and system for data transfer between compute clusters and file system

A data storage system is implemented with an active power monitoring and control performed by a control node elected among a number of nodes. A real-time power monitoring information is supplied to the control node from, a power monitoring logic residing at each device in the system. The devices in the data storage system are pre-allocated with respective individual power budgets which are below the maximum power usage thereof. The power budgets of all the equipment cumulatively constitute a power budget assigned to the group of equipment. The control node controls dynamically and in real time power sharing between the plurality devices so that the devices with required power usage below the pre-allocated power budget can share their extra power credits with devices which are in need for extra power for performing its operation. The control node provides sharing of the power among the equipment in the data storage system with a goal of avoiding exceeding of the cumulative power budget assigned for the entire system or the cluster of the equipment.

Data storage system with active power management and method for monitoring and dynamical control of power sharing between devices in data storage system

The present invention is directed to data migration, and particularly, Parity Group migration, between high performance data generating entities and data storage structure in which distributed NVM arrays are used as a single intermediate logical storage which requires a global registry/addressing capability that facilitates the storage and retrieval of the locality information (metadata) for any given fragment of unstructured data and where Parity Group Identifier and Parity Group Information (PGI) descriptors for the Parity Groups' members tracking, are created and distributed in the intermediate distributed NVM arrays as a part of the non-deterministic data addressing system to ensure coherency and fault tolerance for the data and the metadata. The PGI descriptors act as collection points for state describing the residency and replay status of members of the Parity Groups.

Method and system for data migration between high performance computing architectures and file system using distributed parity group information structures with non-deterministic data addressing

Method and system for reclamation of distributed dynamically generated erasure groups for data migration between high performance computing architectures and data storage using non-deterministic data addressing

Data storage systems and methods for storing data are described herein. The storage system may be integrated with or coupled with a compute cluster or super computer having multiple computing nodes. A plurality of nonvolatile memory units may be included with computing nodes, coupled with computing nodes or coupled with input/output nodes. The input/output nodes may be included with the compute cluster or super computer, or coupled thereto. The nonvolatile memory units store data items provided by the computing nodes, and the input/output nodes maintain where the data items are stored in the nonvolatile memory units via a hash table distributed among the input/output nodes. The use of a distributed hash table allows for quick access to data items stored in the nonvolatile memory units even as the computing nodes are writing large amounts of data to the storage system quickly in bursts.

Data storage architecture and system for high performance computing incorporating a distributed hash table and using a hash on metadata of data items to obtain storage locations

Data storage systems and methods for storing data are described herein. The storage system includes a first storage node is configured to issue a first delivery request to a first set of other storage nodes in the storage system, the first delivery request including a first at least one data operation for each of the first set of other storage nodes and issuing at least one other delivery request, while the first delivery request remains outstanding, the at least one other delivery request including a first commit request for each of the first set of other storage nodes. The first node causes the first at least one data operation to be made active within the storage system in response to receipt of a commit indicator along with a delivery acknowledgement regarding one of the at least one other delivery request.

Maintaining order and fault-tolerance in a distributed hash table system

Data storage systems and methods for storing data are described herein. The storage system includes at least two data storage nodes for storing portions of a distributed hash table and related data. After a first node attempts to complete a write request at a second node and is unable to complete the request, the first node ceases responses to interactions from other nodes. Once the first node's failure to respond has caused a sufficient number of nodes to cease responding, the nodes enter a service mode to resolve the live lock. While in live lock, the nodes determine the oldest, unfulfilled request using a system-wide logical timestamp associated with write requests. Once the oldest request is determined, a removal vote to remove the non-responsive node from the group is initiated and, if other nodes agree, the non-responsive node is removed from the group of nodes.

Handling failed transaction peers in a distributed hash table

Data storage architecuture and system for high performance computing hash on metadata in reference to storage request in nonvolatile memory (NVM) location

Systems and methods for reducing metadata in a write-anywhere storage system are disclosed herein. The system includes a plurality of clients coupled with a plurality of storage nodes, each storage node having a plurality of primary storage devices coupled thereto. A memory management unit including cache memory is included in the client. The memory management unit serves as a cache for data produced by the clients before the data is stored in the primary storage. The cache includes an extent cache, an extent index, a commit cache and a commit index. The movement of data and metadata is by an interval tree. Methods for reducing data in the interval tree increase data storage and data retrieval performance of the system.

Reducing metadata in a write-anywhere storage system

Distributed Compute Engine (DCE) memory controller in a data storage environment contains embedded logic and arithmetic functionality for Boolean logical and arithmetic operations. 'Write' or 'Read' requests which are received from data generating entities, contain a Physical Address field identifying an address of a data block to be retrieved from the external memory, and a Control bits field identifying a type of computational operation to be performed. The DCE memory controller decodes the request, and applies the desired compute operation autonomically to the contents of an external memory and/or the incoming data without burdening the CPU with the computational activity.

Data storage system employing a distributed compute engine memory controller with embedded logic and arithmetic functionality and method for data migration between high-performance computing architectures and data storage devices using the same

Growing community of inventors

Laurel, MD, United States of America

Pavan Kumar Uppu