Algorithms for constructing sets of frequently occurring strings

Abstract

Media and methods are provided for constructing data structures associating data segments in packets with reference identifiers to optimize bandwidth usage within a network during transmission. In one embodiment, a data structure is initialized to contain no data. A minimum length of a data segment to be examined is determined. For each packet in the transmission, data segments of the packet are examined. Data segments are added to the data structure as necessary. Examination is repeated along with an appropriate addition until the packet contains less data than a segment of the minimum length and further continued until all packets are examined. The data structure is constructed associating the portions of the packet and respective reference identifiers. In certain embodiments, the count can be a total count of frequently occurring segments, the count of unique segments or the count of packets containing unique segments. In other embodiments, the construction is accomplished by a compression library. In still other embodiments, one or more filtering criteria are added to the process to optimize efficiency. Embodiments are described where packet size, packet source, packet destination or proportion of alpha-numeric characters are used as the filtering criteria.