Confidence-driven workflow orchestrator for data labeling

Abstract

One embodiment includes a computer-implemented data labeling platform. The platform provides a confidence-driven workflow (CDW) executable to receive and process labeling requests to label data items. The CDW comprises a set of executable labelers, each labeler in having a dynamically modeled confidence range. The execution path for processing a labeling request to label a data item is dynamically determined. Dynamically determining the execution path comprises dynamically determining a bounded number of candidate paths through the set of labelers using dynamically calculated cost and confidence metrics for the labelers in the set of labelers to estimate a probability of each candidate path to satisfy a set of constraints on cost and final result confidence, selecting a candidate path that minimizes cost for a specified confidence from the candidate paths as a selected path, executing a next labeler consultation according to the selected path to label the data item, and dynamically re-determining the remaining execution path using calculated results arising from executing the completed path steps.