Prompt-based language models for generating multi-modal electronic records

Abstract

An example embodiment may involve obtaining text-based, ground truth electronic health records (EHRs), wherein the ground truth EHRs specify a sequence of medical visits involving a plurality of modalities, and wherein each of the medical visits specifies tokens representing at least one of the modalities; generating a training data set by perturbing the ground truth EHRs, wherein perturbing the ground truth EHRs involves deleting or shuffling some of the tokens in the ground truth EHRs; and iteratively applying a machine learning trainer application to the training data set, wherein the machine learning trainer application includes: (i) a bidirectional language model encoder that takes EHRs within the training data set and produces vector embeddings therefrom, (ii) an autoregressive language model decoder that takes the vector embeddings and infers predicted EHRs therefrom, and (iii) a loss function that compares the predicted EHRs to their corresponding ground truth EHRs.