Machine learning-based url categorization system with noise elimination

Abstract

The technology discloses training a classifier to label webpages with categories, extracting from a training database with thousands of webpages tentatively labeled with ground truth categories, dataset A and dataset B; training a classifier using A; and applying it to webpages in B to assign a webpage a label and a classification score. Also disclosed is cleaning B, removing webpages based on evaluation of a decision confidence metric derived from the score assigned for the webpage; and training a second classifier using cleaned B, with second classifier weights initialized independent of trained first classifier weights; and applying the second classifier to webpages in A to assign a webpage the label, score, and decision confidence matrix, and cleaning A, removing webpages from A based on the decision confidence metric. Then combining cleaned A and cleaned B into a combined clean dataset, and training the third classifier using the combined clean dataset.