Method and apparatus for accelerating deduplication processing

Abstract

A deduplication engine maintains a hash table containing hash values of tracks of data stored on managed drives of a storage system. The deduplication engine keeps track of how frequently the tracks are accessed by the deduplication engine using an exponential moving average for each track. Target tracks which are frequently accessed by the deduplication engine are cached in local memory, so that required byte-by-byte comparisons between the target track and write data may be performed locally rather than requiring the target track to be read from managed drives. The deduplication engine implements a Least Recently Used (LRU) cache data structure in local memory to manage locally cached tracks of data. If a track is to be removed from local memory, a final validation of the target track is implemented on the version stored in managed resources before evicting the track from the LRU cache.