Method for providing high performance scalable file i/o through persistent file domain and functional partitioning

Abstract

A method for implementing large scale parallel file I/O processing includes steps of: separating processing nodes into compute nodes specializing in computation and I/O nodes (computer processors restricted to running I/O daemons); organizing the compute nodes and the I/O nodes into processing sets, the processing sets including: one dedicated I/O node corresponding to a plurality of compute nodes. I/O related system calls are received in the compute nodes then sent to the corresponding I/O nodes. The I/O related system calls are processed through a system I/O daemon residing in the I/O node. The plurality of compute nodes are evenly distributed across participating processing sets. Additionally, for collective I/O operations, compute nodes from each processing set are assigned as I/O aggregators to issue I/O requests to their corresponding I/O node, wherein the I/O aggregators are evenly distributed across the processing set. Additionally, a file domain is partitioned using a collective buffering technique wherein data is aggregated in memory prior to writing to a file; portions of the partitioned file domain are assigned to the processing sets.