Asymmetric end host redundancy elimination for networks

Abstract

An end host redundancy elimination system and method to provide redundancy elimination as an end system service. Embodiments of the system and method use optimization techniques that reduce server central processing unit (CPU) load and memory footprint as compared to existing approaches. For server storage, embodiments of the system and method use a suite of highly-optimized data structures for managing metadata and cached payloads. An optimized asymmetric max-match technique exploits the inherent structure in data maintained at the server and client and ensures that client processing load is negligible. A load-adaptive fingerprinting technique is used that is much faster than current fingerprinting techniques while still delivering similar compression. Load-adaptive means that embodiments of the fingerprinting technique can adapt CPU usage depending on server load. Embodiments of the system and method operate above the transmission control protocol (TCP) layer, thereby reducing the number of roundtrips needed for data transfer.