Image classification with modality dropout

Abstract

Systems and methods are provided for classifying images associated with an item, and generating an image set for that item which includes image classifications determined to be helpful for the item type of the item. To classify images, an image classification model is generated and trained using two phases. The first phase uses intermediate model with text and visual processing to teach the model to recognize patterns created by text without requiring OCR at inference. The second phase uses visual processing to refine the model for use at inference. To generate an image set, image classifications helpful to an item type are identified, items are associated with item types, images are obtained for an item, the images are classified using the image classification model, missing image classifications set out in the preferred image set are identified, and a request or requests is generated for the missing image classifications.