Generating natural language representations of mental content from functional brain images

Abstract

By way of introduction, the present embodiments described below include apparatuses and methods for generating natural language representations of mental content from functional brain images. Given functional imaging data acquired while a subject reads a text passage, a reconstruction of the text passage is produced. Linguistic semantic vector representations are assigned () to words, phrases or sentences to be used as training stimuli. Basis learning is performed (), using brain imaging data acquired () when a subject is exposed to the training stimuli and the corresponding semantic vectors for training stimuli, to learn an image basis directly. Semantic vector decoding () is performed with functional brain imaging data for test stimuli and using the image basis to generate a semantic vector representing the test imaging stimuli. Text generation () is then performed using the decoded semantic vector representing the test imaging stimuli.