Domain-specific human-model collaborative annotation tool

Abstract

A human-model collaborative annotation system for training human annotators includes a database that stores images previously annotated by an expert human annotator and/or a machine learning annotator, a display that displays images selected from the database, an annotation system that enables human annotators to annotate images presented on the display, and an annotation training system. The annotation training system selects an image sample from the database for annotation by a human annotator, receives one or more proposed annotations from the annotation system, compares the human annotator's one or more proposed annotations to previous annotations of the image sample by the expert human annotator or machine learning annotator, presents attention maps on the display to draw the human annotator's attention to any annotation errors identified by the comparing, and selects a next training image sample from the database based on any errors identified in the comparing step.