Method and system of content based dynamic data compression

Abstract

The present disclosure describes methods and systems that provide for content based dynamic data compression. During a training stage a plurality of training sets are compressed using multiple data compression techniques, and each training data set is associated with a most effective compression technique. Statistical features can be extracted from each of the training data sets with consideration of the most effective compression techniques and are saved as a compression parameters file. During a working stage, individual data blocks within a telemetry data set are compressed using a data compression technique selected for the data block in light of the statistical feature presented in the compression parameters file, and the compressed data blocks are written to a compressed data set, along with encoding tags that identify the compression technique used on each data block. During a consumption stage, the compressed data blocks are decompressed using techniques identified by the encoding tags.