The patent badge is an abbreviated version of the USPTO patent document. The patent badge does contain a link to the full patent document.
The patent badge is an abbreviated version of the USPTO patent document. The patent badge covers the following: Patent number, Date patent was issued, Date patent was filed, Title of the patent, Applicant, Inventor, Assignee, Attorney firm, Primary examiner, Assistant examiner, CPCs, and Abstract. The patent badge does contain a link to the full patent document (in Adobe Acrobat format, aka pdf). To download or print any patent click here.
Patent No.:
Date of Patent:
Apr. 12, 2016
Filed:
Jul. 09, 2012
Joshua M. Attenberg, Roxbury, CT (US);
Panagiotis G. Ipeirotis, New York, NY (US);
Foster J. Provost, New York, NY (US);
Joshua M. Attenberg, Roxbury, CT (US);
Panagiotis G. Ipeirotis, New York, NY (US);
Foster J. Provost, New York, NY (US);
Integral Ad Science, Inc., New York, NY (US);
New York University, New York, NY (US);
Abstract
Methods, systems, and media for identifying errors in predictive models using annotators are provided. In some embodiments, a method for evaluating predictive models in classification systems is provided, the method comprising: causing an input region to be presented to a user, where the input region receives an instance from the user that corresponds to a predictive model; retrieving a classification conducted by the predictive model for the received instance and a confidence value associated with the classification; determining whether the received instance has been incorrectly classified by the predictive model; determining a reward associated with the incorrect classification made by the predictive model in response to determining that the received instance has been incorrectly classified by the predictive model, where the reward is based on the confidence value associated with the classification of the received instance; and providing the reward to the user.