Approach for a controllable trade-off between cost and availability of indexed data in a cloud log aggregation solution such as splunk or sumo

Abstract

Systems and methods for classification of log data at the source into sets of critical and non-critical data. Critical data may be indexed and processed normally, while non-critical data may be provided to and stored by the cloud-based log aggregation system without indexing, at significantly lower cost in terms of processing and storage. In the event that non-critical data is required for troubleshooting or analysis, the non-critical data may be indexed dynamically on request. Because the non-critical data is stored at the cloud-based log aggregation system, it may be quickly indexed and added to the critical data, without additional consumption of bandwidth or delays due to transmission. Dynamic selection and classification of critical and non-critical data may thus allow an enterprise to balance costs and availability of indexed data.