System and method for generating a classification model with a cost function having different penalties for false positives and false negatives

Abstract

Embodiments of a system and method for generating a classification model with a cost function having different penalties for false positives and false negatives are described. Embodiments may include perform machine learning operations on known duplicates and known non-duplicates to generate a classification model for classifying structured data items as duplicates or non-duplicates. Each duplicate may represent a pair of structured data items describing a common item; each non-duplicate may represent a pair of structured data items describing different items. Generation of the classification model may be performed based on a cost function that penalizes false positive misclassifications within the classification model differently than false negative misclassifications. Embodiments may also include evaluating the classification model to determine whether a candidate structured data item is a duplicate or non-duplicate. The classification model may include but is not limited to support vector machines and boosted decision trees.