Scalable, distributed, asynchronous data collection mechanism

Abstract

The “scan” phase of a distributed data collection process is decoupled from upload of the return collection data, with the “scan” consisting merely of an infrequent profile push to configure autonomous scanners at the data collection endpoints. Distributed data collection is initiated by endpoints within the distributed network, which autonomously perform a scan and transmit a Collection Table of Contents (CTOC) data structure to a nearest available collector, then await a ready message from the collector. When ready to receive the return collection data, the collector signals the endpoint, which transfer the data collection in small packets to the collector. The collector stores the received data collection in persistent storage, then initiates collection to a higher collector or recipient in substantially the same manner as the endpoint. A routing manager controls the routing of data from endpoints through one or more collectors to the recipient. Scans for the data collection may thus be performed fully parallel, and upload of the collection data proceeds by direct channel under the control of the collectors. Bandwidth utilization for the data collection may thus be optimized for network loading by blackout periods and cooperation of the collectors with other distributed applications. The resulting distributed data collection mechanism is scalable, with large numbers of endpoints and large return collection data sizes being efficiently supported.