Concurrent data entropy and digest computation operating on same data for cpu cache efficiency

Abstract

Techniques for CPU cache efficiency may include performing concurrent processing, such as for first and second data operations, in a synchronized manner that prevents loading the same data chunk into the CPU cache more than once. Processing may include synchronizing the first and second data operations with respect to a first data chunk to ensure that both complete prior to proceeding with performing such processing on a second data chunk. The first and second data operations for a chunk may be deduplication digest computation and entropy computation performed inline as part of the data path. If the chunk cannot be deduplicated, the entropy value may be used with an adaptive entropy threshold to determine whether to compress the chunk inline. The entropy value may be determined using less than all bytes in the chunk. The chunk's entropy value may be determined based on a data set entropy value.