Feature equivalence and document abnormality threshold determination

Abstract

A method includes, as part of establishing a feature merging threshold (α) for determining equivalence between two features, selecting a set of candidate α values, partitioning training data into a plurality of groups, establishing a model Wfor each α value of the set of candidate α values, iteratively performing: selecting a next group of training data of the plurality of groups of training data; adding the selected next group of training data to a training set; and for each α value in the set of candidate α values: training the Wfor the α value using the training set, and evaluating a size of W, the size comprising a number of features included in the model, and choosing the feature merging threshold α based on the iteratively performing.