Network file deduplication using decaying bloom filters

Abstract

A system for receiving and deduplicating data strings transmitted over a network is disclosed. The system comprises one or more network sensors detecting data strings while in transit on the network; and non-transitory memory comprising instructions. When the instructions are executed by one or more processors, the one or more processors establish a plurality of Bloom filters, receive a first data string, perform a first insertion operation into each Bloom filter; determine, for each of one or more Bloom filters, a set of bits, whether presently set or cleared, to be unset; and unset each determined set of bits in the one or more Bloom filters. At a later moment in time, the first data string is received again, and each Bloom filter is queried to determine whether the first data string has been inserted, based on a current state of that Bloom filter.