Voicebox Technologies Corporation

Other

Stephen Steele Carter

Daniela Braga

Ahmad Khamis Elshenawy

Spencer John Rothwell

Faraz Romani

Michael Kennewick

A method includes causing a first crowdsourced validation job to be provided to one or more first validation devices, the first crowdsourced validation job comprising first instructions for a crowd user to provide an indication of an accuracy of a transcription of natural language content, receiving a plurality of responses from the one or more first validation devices, wherein the plurality of responses include at least a first response from at least a first validation device from among the one or more first validation devices, the first response including a first indication of an accuracy of the transcription of the natural language content, and determining a first confidence score of the first validation device based, at least in part, on the plurality of responses received from the one or more first validation devices and the first response received from the first validation device.

System and method for validating natural language content using crowdsourced validation jobs

Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A group of validation devices may be selected for reviewing the transcription pair. A crowdsourced validation job may be created for the group of validation devices. The crowdsourced validation job may be provided to the group of validation devices. One or more votes representing whether or not the text accurately represents the natural language content may be received from the group of validation devices. Based on the one or more votes received, the transcription pair may be stored in a validated transcription library, which may be used to process end-user voice data.

A system and method of tagging utterances with Named Entity Recognition ('NER') labels using unmanaged crowds is provided. The system may generate various annotation jobs in which a user, among a crowd, is asked to tag which parts of an utterance, if any, relate to various entities associated with a domain. For a given domain that is associated with a number of entities that exceeds a threshold N value, multiple batches of jobs (each batch having jobs that have a limited number of entities for tagging) may be used to tag a given utterance from that domain. This reduces the cognitive load imposed on a user, and prevents the user from having to tag more than N entities. As such, a domain with a large number of entities may be tagged efficiently by crowd participants without overloading each crowd participant with too many entities to tag.

System and method of annotating utterances based on tags assigned by unmanaged crowds

Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.

System and method for eliciting open-ended natural language responses to questions to train natural language processors

A system and method of recording utterances for building Named Entity Recognition ('NER') models, which are used to build dialog systems in which a computer listens and responds to human voice dialog. Utterances to be uttered may be provided to users through their mobile devices, which may record the user uttering (e.g., verbalizing, speaking, etc.) the utterances and upload the recording to a computer for processing. The use of the user's mobile device, which is programmed with an utterance collection application (e.g., configured as a mobile app), facilitates the use of crowd-sourcing human intelligence tasking for widespread collection of utterances from a population of users. As such, obtaining large datasets for building NER models may be facilitated by the system and method disclosed herein.

System and method of recording utterances using unmanaged crowds for natural language processing

Systems and methods of validating transcriptions of natural language content using crowdsourced validation jobs are provided herein. In various implementations, a transcription pair comprising natural language content and text corresponding to a transcription of the natural language content may be gathered. A first group of validation devices may be selected for reviewing the transcription pair. A first crowdsourced validation job may be created for the first group of validation devices. The first crowdsourced validation job may be provided to the first group of validation devices. A vote representing whether or not the text accurately represents the natural language content may be received from each of the first group of validation devices. A validation score may be assigned to the transcription pair based, at least in part, on the votes from each of the first group of validation devices.

Growing community of inventors

Seattle, WA, United States of America

Stephen Steele Carter