Crowdsourced training of textual natural language understanding systems

Abstract

A facility to crowdsource training of virtual assistants and other textual natural language understanding systems is described. The facility first specifies a set of possible user intents (e.g., a kind of question asked by users). As part of specifying an intent, entities, that represent salient items of information associated with the intent are identified. Then, for each of the intents, the facility directs users of a crowdsourcing platform to input a number of different textual queries they might use to express this intent. Then, additional crowdsourcing platform users are asked to perform semantic annotation of the cleaned queries, for each selecting its intent and entities from predefined lists. Next, still other crowdsourcing platform users are asked whether the selection of intents and entities during semantic annotation was correct for each query. Once validated, the annotated queries are used to train the assistant.