Datadirect Networks, Inc.

Michael Piszczek

Jason Micah Cope

Pavan Kumar Uppu

Paul Joseph Nowoczynski

William Joseph Harker

John Gordon Manning

Cedric T Fernandes

Thomas E Fugini

David F Fellinger

Gordon Manning

Lee Douglas McBryde

Donald Joseph Molaro

Manjari Mishra

Don Molaro

Shivkumar Pandit

J Gordon Manning

Rajkumar Joshi

Dave Fellinger

Lauren Belella

Zhiwei Sun

John Albert De Leon

Charles Dwain Cole, Jr

Pavan Kumar Kumar Uppu

Thomas Pontos

Dave Illar

Richard Williams

Andrew Pershall

The present invention is directed to a distributed data migration and storage system and method configured for fully parallel writing and retrieval of data objects to (and from) the Web Object Scaler (WOS) object storage system with highly efficient accessibility and using a simplified set of filing commands. The data objects migrate between data generating entities and the WOS object storage through an intermediate Burst Buffer tier (supported by the Infinite Memory Engine) without the utilization of a serial process substantially simultaneously through parallel channels. The metadata for each object is captured with appropriate pointers for efficient finding of the object in the WOS storage system. The WOS nodes return the stored objects' ID (OIDs) corresponding to the objects storage address which are used for the object access.

Method and system for parallel file operation in distributed data storage system with mixed types of storage media

A data migrating system and method are provided in which a Burst Buffer Network Aggregator (BBNA) process is configured either on the File Servers or on the File System's dedicated I/O nodes to coalesce data fragments stored in participating Burst Buffer nodes under the direction of a primary BB node appointed by a data generating entity prior to transfer of the full data stripe into the File System. The 'write' request in the form of a full data stripe is distributed into a plurality of data fragments among participating BB nodes along with corresponding metadata. The primary BB node gathers the metadata from the participating BB nodes, sends the metadata list to the BBNA unit, responsive to which the BBNA unit allocates a buffer sufficient to store the full data stripe, and transfers data fragments from participating BB nodes into the full data stripe buffer, thereby coalescing the data fragments into the full data stripe, which is subsequently transferred from the buffer in the BBNA unit into the File System.

Method for data transfer between compute clusters and file system

A data migration system supports a low-latency and reduced overhead data storage protocol for data storage sharing in a non-collision fashion which does not require inter-communication and permanent arbitration between data storage controllers to decide on the data placement/routing. The multiple data fragments of data sets are prevented from routing to the same storage devices by a multi-step selection protocol which selects (in a first phase of the selection routine) a healthy highest ranked drive enclosure, and further selects (in a second phase of the selection routine) a healthy highest-ranked data storage controller residing in the selected drive enclosure, for routing data fragments to different storage pools assigned to the selected data storage devices for exclusive 'writing' and data modification. The selection protocol also contemplates various failure scenarios in a data placement collision free manner.

Low latency and reduced overhead data storage system and method for sharing multiple storage devices by high performance computing architectures

The system and routine for data caching leverages the properties of Network-Attached Non-Volatile Memories (NANVMs) to provide virtualized secure node-local storage services to the network users with reduced data movement across the NANVMs. The caching routine reserves storage resources (storage partitions) on NANVM devices, migrates data required for the target application execution to the allocated storage partitions, and directs the network clients to dynamically 'mount' to the storage partitions based on application data requirements. Only those clients and applications that present valid credentials and satisfactory computing capabilities can access the data in the specific storage partitions. Several clients can have an access to the same storage partitions without duplication or replicating the data. A Global Data Indexing sub-system supports the efficient operation of the subject system. The Global Data Indexing Sub-System provides mapping between the storage partitions, data sets, applications, client nodes, as well as their credentials/capabilities.

System and method for scale-out node-local data caching using network-attached non-volatile memories

Variable Redundancy Distributed (VRD) RAID controller in a data storage environment contains embedded RAID logic permitting to choose and compute a desired redundancy coding scheme from a plurality thereof pre-programmed and embedded in a Compute Engine in the VRD RAID controller. 'Write' or 'Read' requests which are received from data generating entities, contain information identifying a type of the redundancy coding scheme of interest. The controller decodes the request, and automatically applies the desired computation to the incoming data without burdening the CPU with the computational activity. The variable redundancy computational ability of the subject systems provides an extremely versatile and flexible tool for RAID operations.

Data storage system employing a variable redundancy distributed RAID controller with embedded RAID logic and method for data migration between high-performance computing architectures and data storage devices using the same

Method and system for data migration between data generating entities and data storage devices protected by de-clustered RAID algorithm are enhanced by dynamically controlling the I/O activity towards the data storage devices (NVM devices) based on their remaining lifespan (health) with the goal to prevent multiple devices selected for writing a parity stripe information from simultaneous failures. This feature is rendered by polling the remaining health of NVM devices in the RAID pool, computing a weighted lifespan for each NVM device, comparing the latter to an average of all NVM devices in the pool, and adjusting the I/O activity towards the NVM device of interest accordingly. If the weighted lifespan exceeds the average lifespan in the pool, the allowed I/O activity is increased, and if the weighted lifespan is below the average for the pool, then the device in question is sent less 'writes'.

Method and system for data migration between high performance computing entities and a data storage supported by a de-clustered raid (DCR) architecture with I/O activity dynamically controlled based on remaining health of data storage devices

In the data storage system the storage area network performs XOR operations on incoming data for parity generation without buffering data through a centralized RAID engine or processor. The hardware for calculating the XOR data is distributed to incrementally calculate data parity in parallel across each data channel and may be implemented as a set of FPGAs with low bandwidths to efficiently scale as the amount of storage memory increases. A host adaptively appoints data storage controllers in the storage area network to perform XOR parity operations on data passing therethrough. The system provides data migration and parity generation in a simple and effective matter and attains a reduction in cost and power consumption.

Data storage system and method for data migration between high-performance computing architectures and data storage devices using storage controller with distributed XOR capability

Method and system for data transfer between compute clusters and file system

A data storage system is implemented with an active power monitoring and control performed by a control node elected among a number of nodes. A real-time power monitoring information is supplied to the control node from, a power monitoring logic residing at each device in the system. The devices in the data storage system are pre-allocated with respective individual power budgets which are below the maximum power usage thereof. The power budgets of all the equipment cumulatively constitute a power budget assigned to the group of equipment. The control node controls dynamically and in real time power sharing between the plurality devices so that the devices with required power usage below the pre-allocated power budget can share their extra power credits with devices which are in need for extra power for performing its operation. The control node provides sharing of the power among the equipment in the data storage system with a goal of avoiding exceeding of the cumulative power budget assigned for the entire system or the cluster of the equipment.

Data storage system with active power management and method for monitoring and dynamical control of power sharing between devices in data storage system

The present invention is directed to data migration, and particularly, Parity Group migration, between high performance data generating entities and data storage structure in which distributed NVM arrays are used as a single intermediate logical storage which requires a global registry/addressing capability that facilitates the storage and retrieval of the locality information (metadata) for any given fragment of unstructured data and where Parity Group Identifier and Parity Group Information (PGI) descriptors for the Parity Groups' members tracking, are created and distributed in the intermediate distributed NVM arrays as a part of the non-deterministic data addressing system to ensure coherency and fault tolerance for the data and the metadata. The PGI descriptors act as collection points for state describing the residency and replay status of members of the Parity Groups.

Method and system for data migration between high performance computing architectures and file system using distributed parity group information structures with non-deterministic data addressing

Systems and methods for reducing metadata in a write-anywhere storage system are disclosed herein. The system includes a plurality of clients coupled with a plurality of storage nodes, each storage node having a plurality of primary storage devices coupled thereto. A memory management unit including cache memory is included in the client. The memory management unit serves as a cache for data produced by the clients before the data is stored in the primary storage. The cache includes an extent cache, an extent index, a commit cache and a commit index. The movement of data and metadata is by an interval tree. Methods for reducing data in the interval tree increase data storage and data retrieval performance of the system.

Reducing metadata in a write-anywhere storage system

Method and system for reclamation of distributed dynamically generated erasure groups for data migration between high performance computing architectures and data storage using non-deterministic data addressing

Data storage systems and methods for storing data are described herein. The storage system may be integrated with or coupled with a compute cluster or super computer having multiple computing nodes. A plurality of nonvolatile memory units may be included with computing nodes, coupled with computing nodes or coupled with input/output nodes. The input/output nodes may be included with the compute cluster or super computer, or coupled thereto. The nonvolatile memory units store data items provided by the computing nodes, and the input/output nodes maintain where the data items are stored in the nonvolatile memory units via a hash table distributed among the input/output nodes. The use of a distributed hash table allows for quick access to data items stored in the nonvolatile memory units even as the computing nodes are writing large amounts of data to the storage system quickly in bursts.

Data storage architecture and system for high performance computing incorporating a distributed hash table and using a hash on metadata of data items to obtain storage locations

Data storage systems and methods for storing data are described herein. The storage system includes a first storage node is configured to issue a first delivery request to a first set of other storage nodes in the storage system, the first delivery request including a first at least one data operation for each of the first set of other storage nodes and issuing at least one other delivery request, while the first delivery request remains outstanding, the at least one other delivery request including a first commit request for each of the first set of other storage nodes. The first node causes the first at least one data operation to be made active within the storage system in response to receipt of a commit indicator along with a delivery acknowledgement regarding one of the at least one other delivery request.

Maintaining order and fault-tolerance in a distributed hash table system

Data storage systems and methods for storing data are described herein. The storage system includes at least two data storage nodes for storing portions of a distributed hash table and related data. After a first node attempts to complete a write request at a second node and is unable to complete the request, the first node ceases responses to interactions from other nodes. Once the first node's failure to respond has caused a sufficient number of nodes to cease responding, the nodes enter a service mode to resolve the live lock. While in live lock, the nodes determine the oldest, unfulfilled request using a system-wide logical timestamp associated with write requests. Once the oldest request is determined, a removal vote to remove the non-responsive node from the group is initiated and, if other nodes agree, the non-responsive node is removed from the group of nodes.

Growing community of inventors

Laurel, MD, United States of America

Michael Piszczek