The patent badge is an abbreviated version of the USPTO patent document. The patent badge does contain a link to the full patent document.
The patent badge is an abbreviated version of the USPTO patent document. The patent badge covers the following: Patent number, Date patent was issued, Date patent was filed, Title of the patent, Applicant, Inventor, Assignee, Attorney firm, Primary examiner, Assistant examiner, CPCs, and Abstract. The patent badge does contain a link to the full patent document (in Adobe Acrobat format, aka pdf). To download or print any patent click here.
Patent No.:
Date of Patent:
Jan. 28, 2020
Filed:
Mar. 15, 2019
International Business Machines Corporation, Armonk, NY (US);
Masaki Komedani, Kanagawa-ken, JP;
Ken Kumagai, Tokyo, JP;
Takuma Murakami, Tokyo, JP;
Akihiro Nakayama, Kawasaki, JP;
International Business Machines Corporation, Armonk, NY (US);
Abstract
A method for evaluating annotation quality is provided. The method may include obtaining annotation information associated with a plurality of annotators and a plurality of data elements including a plurality of annotation entries corresponding to at least one data element and entered based on an annotation guideline, determining a quality rating for the annotation guideline based on a comparison between a first value associated with the plurality of annotators and the plurality of data elements and a second value associated with any disparity among the plurality of annotation entries, determining a proficiency rating for an annotator from the plurality of annotators based on a comparison between a third value associated with annotation entries by the annotator and the second value, and generating a report based on the quality rating and the proficiency rating.