Method and process for enabling distributing cache data sources for query processing and distributed disk caching of large data and analysis requests

Abstract

A method and system for large data and distributed disk cache processing in a Pneuron platform. The system and method include three specific interoperable but distributed functions: the adapter/cache Pneuronand distributed disk files, a dynamic memory mapping tree, and distributed disk file cleanup. The system allows for large data processing considerations and the ability to access and acquire information from large data setsand rapidly distribute and provide the information to subsequent Pneuronsfor processing. The system also provides the ability to store large result sets, the ability to deal with sequential as well as asynchronous parallel processing, the ability to address large unstructured data; web logs, email, web pages, etc., as well as the ability to handle failures to large block processing.