Automatic evaluation of categorization system quality

Abstract

A computerized method and system of document analysis. The method and system categorise documents according to a taxonomy. This is accomplished by rating training documents on a lower level by associating either of the following predicates to a training document: either correct, inbound, outbound, or unassigned, Rating categories are established on a lower level by determining precision/recall values for each category, and generating higher level category rating attributes from the lower-level rating steps. This is done by associating one or more of: aa) weak category, bb) existing source/sink relationship between categories, cc) close categories to the categories, and deriving an overall quality measure for the training base from the lower-level and higher-level rating step. The lower-level and higher-level evaluation results are stored. The quality measure is used to determine action proposals to improve the training base as either one or more of: aa) modifying the number of categories by adding a new category or deleting an existing category, or bb) splitting a category in one or more new categories, or cc) merging a category with another one, or dd) modifying the number of training documents of a category by adding or removing some of them, and optionally, providing a means to automatically carry out the above steps and review the results including the ability to restore the previous state.