Sampling content using machine learning to identify low-quality content

Abstract

An online system obtains risk scores determined by a machine learning model for a content item provided by a user of an online system for display to users of the online system, where the risk scores indicate the likelihood of content items violating a content policy. The online system uses the risk scores to determine sampling weights used to select content items for inclusion in a sampled subset of content items. The sampling weights are determined from risk score counts indicating the relative frequency of the obtained risk scores and impression counts indicating the number of times content items have been presented to the users of the online system. The online system presents the selected content items for evaluation by a human reviewer using a quality review interface. Using the results of the quality review, the online system determines quality performance metrics of the machine learning model.