Automated data generation, post-job validation and agreement voting using automatic results generation and humans-in-the-loop, such as for tasks distributed by a crowdsourcing system

Abstract

Systems and methods that use machine learning to optimize the execution of micro-tasks, by partially automating the generation and validation actions are disclosed. The system uses a combination of automatic intelligent model-based decision systems and human-in-the-loop for generating annotated task instances with respect to an identified task. Before a task is executed, the system can compute the crowd effort to generate data for each task instance, as well as the effort to validate and/or correct them. These computations can occur multiple times during the execution of task. The generation, validation and correction effort are measures that allow the system to design more efficient workflows that combine machine learning models and human input because the system can decide automatically what is the most efficient next step to obtain the best results.