Data compression with redundancy removal across boundaries of compression search engines

Abstract

Data compression techniques are provided that remove redundancy across the boundary of compression search engines. An illustrative method comprises splitting the data frame into a plurality of sub-chunks; comparing at least two of the plurality of sub-chunks to one another to remove at least one sub-chunk from the plurality of sub-chunks that substantially matches at least one other sub-chunk to generate a remaining plurality of sub-chunks; generating matching sub-chunk information for data reconstruction identifying the at least one removed sub-chunk and the corresponding substantially matched at least one other sub-chunk; grouping the remaining plurality of sub-chunks into sub-units; removing substantially repeated patterns within the sub-units to generate corresponding compressed sub-units; and combining the compressed sub-units with the matching sub-chunk information to generate a compressed data frame. The data frame optionally comprises one or more host pages compressed substantially simultaneously, and the compressed data frame for a plurality of host pages compressed substantially simultaneously comprises a host page address for each host page.