Random and active learning for classifier training

Abstract

An online system actively and randomly selects content items to be labeled for training a classifier. An online system receives content items from client devices of users and selects sets of the content items to be labeled by human labelers. The randomly selected content items are selected at random from the received content items, and the actively selected content items are selected based on the classifier's confidence in accurately predicting the classification of the content items. The online system may use a histogram of content items to actively select content items. The online system assigns the content items to bins of the histogram based on priority scores and selects content items with priority scores of the highest percentile. The online system provides the selected content items to human labelers for labeling. The labeled content items are then used for training the classifier.