International Business Machines Corporation

Assaf Arbelle

Amit Alfassy

Leonid Karlinsky

Joseph Shtok

Eliyahu Schwartz

Sivan Doveh

An example system includes a processor to automatically extract text and images from a document. The processor can automatically generate text bags including a number of nearest texts for each of the extracted images. The processor can then train a multi-modal model based on the automatically generated text bags using a CLIP-MIL loss that computes, for each of the extracted images, a correlation between each of the different texts in the texts bags using a CLIP feature space at each gradient step of the gradient descent-based multiple instance learning (MIL) algorithm.

Training multi-modal models on documents using multiple instance learning

An example system includes a processor that can randomly mask tokens using different masks to generate different subsets of masked tokens. The processor can process the different sets of masked tokens via a pretrained masked auto-encoder (MAE) encoder to output intermediate representations. The processor can process the intermediate representations via a pretrained MAE decoder to output reconstructed images. The processor can further compare input image with the output reconstructed images to generate an anomaly score.

Anomaly detection using masked auto-encoder

An example system includes a processor to receive, a randomly generated alpha-map, a pair of training images, and a pair of training texts associated with the pair of training images. The processor is to generate a blended image based on the randomly generated alpha-map and the pair of training images. The processor is to train a visual language grounding model to separate the blended image into a pair of heatmaps identifying portions of the blended image corresponding to each of the training images using a separation loss.

Growing community of inventors

Lehvot Haviva, Israel

Assaf Arbelle