Harvesting question/answer training data from watched hypotheses in a deep qa system

Abstract

Embodiments can provide a computer implemented method for harvesting training data for a training set for use by a system capable of answering questions, the system comprising a processor and a memory comprising instructions executed by the processor, the method comprising receiving, from a user, an input question; processing the input question and returning, to the user, a result set comprising one or more ranked hypotheses and one or more ranked evidence passages corresponding to the one or more ranked hypotheses; receiving, from the user, an indication that one of the one or more ranked hypotheses is to be designated a watched hypothesis; adding the input question and the watched hypothesis to a to-be-vetted question/answer (QA) pair set comprising one or more to-be-vetted QA pairs; vetting each of the one or more to-be-vetted QA pairs in the to-be-vetted QA pair set through a first-pass automatic vetting procedure; if a vetted QA pair passes the first-pass automatic vetting procedure, adding the vetted QA pair to the training set; and retraining one or more ranking models based upon the training set.