System for performing collective symbol-based compression of a corpus of

Abstract

A method and apparatus for compressing a corpus of document images into a collective tokenized representation. Initially, documents in the corpus are individually compressed into a document tokenized format. A document image in the document tokenized format is represented using a symbol table and a table of positions. Each symbol in the symbol table is a shape in the original document image. The positions in the table of positions indicates where the symbols in the symbol table are placed to form the document image. Subsequently, the individual symbol tables of each document in the corpus are assembled to form clusters of similar shapes. These clusters are then analyzed to identify the degree of interrelationship between the symbols in the individual symbol tables. Individual document symbol tables with a large number of recurring symbols are grouped together. For each of the groups of symbol tables, a collective symbol table is computed. The collective symbol table improves the compression ratio of a corpus by eliminating redundant shapes appearing in the individual document symbol tables. Also, the collective symbol table advantageously identifies groupings of documents in the corpus which are related because a significant number of similar shapes are used in each of the documents.