System to determine sentiment from audio data

Abstract

A device with a microphone acquires audio data of a user's speech. A neural network accepts audio data as input and provides sentiment data as output. The neural network is trained using training data based on input from raters who provide votes as to which sentiment descriptors they think are associated with a sample of speech. A vote by a rater assessing the sample for a particular semantic descriptor is distributed to a plurality of semantically similar semantic descriptors. Semantic descriptor similarity data indicates relative similarity between possible semantic descriptors in the semantic space. The distributed partial votes may be aggregated to produce training data comprising samples of speech and weights of corresponding semantic descriptors. The training data is then used to train the neural network. For example, the neural network may be trained with the training data using per-instance cosine similarity loss or correlational loss.