International Business Machines Corporation

Amit Alfassy

Joseph Shtok

Leonid Karlinsky

Assaf Arbelle

Amit Aides

Sivan Doveh

An example system includes a processor to automatically extract text and images from a document. The processor can automatically generate text bags including a number of nearest texts for each of the extracted images. The processor can then train a multi-modal model based on the automatically generated text bags using a CLIP-MIL loss that computes, for each of the extracted images, a correlation between each of the different texts in the texts bags using a CLIP feature space at each gradient step of the gradient descent-based multiple instance learning (MIL) algorithm.

Training multi-modal models on documents using multiple instance learning

An example system includes a processor to receive, a randomly generated alpha-map, a pair of training images, and a pair of training texts associated with the pair of training images. The processor is to generate a blended image based on the randomly generated alpha-map and the pair of training images. The processor is to train a visual language grounding model to separate the blended image into a pair of heatmaps identifying portions of the blended image corresponding to each of the training images using a separation loss.

Training visual language grounding models using separation loss

A system for generating a set of digital image features, comprising at least one hardware processor adapted for: producing a plurality of input groups of features, each produced by extracting a plurality of features from one of a plurality of digital images; computing an output group of features by inputting the plurality of input groups of features into at least one prediction model trained to produce a model group of features in response to at least two groups of features, such that a model set of labels indicative of the model group of features is similar, according to at least one similarity test, to a target set of labels computed by applying at least one set operator to a plurality of input sets of labels each indicative of one of the at least two groups of features; and providing the output group of features to at least one other processor.

Growing community of inventors

Haifa, Israel

Amit Alfassy