International Business Machines Corporation

Elad Amrani

Udi Barzelay

Rami Ben-Ari

Tal Hakim

Daniel Nechemia Rotman

Dror Porat

An example system includes a processor to receive unannotated multimodal data. The processor can estimate a probability an associated pair of different modalities in the unannotated multimodal data to be correctly associated using a multimodal similarity function and a local density estimation. The processor can also train a multimodal representation learning model on the unannotated multimodal data using the estimated probability as a weight for the associated pair in a loss function.

Training multimodal representation learning model on unannotated multimodal data

An example system includes a processor to receive image samples for training. The processor can generate different augmented views of each of the image samples. The processor can then train a self-classifier neural network using the different augmented views to minimize a cross entropy of the different augmented views in which a uniform prior is asserted on class predictions.

Training self-classifier using image augmentations and uniform prior

An example system includes a processor to receive an image containing an object to be detected. The processor is to detect the object in the image via a binary object detector trained via a self-supervised training on raw and unlabeled videos.

Object detector trained via self-supervised training on raw and unlabeled videos

An example system includes a processor to receive raw and unlabeled videos. The processor is to extract speech from the raw and unlabeled videos. The processor is to extract positive frames and negative frames from the raw and unlabeled videos based on the extracted speech for each object to be detected. The processor is to extract region proposals from the positive frames and negative frames. The processor is to extract features based on the extracted region proposals. The processor is to cluster the region proposals and assign a potential score to each cluster. The processor is to train a binary object detector to detect objects based on positive samples randomly selected based on the potential score.

Training an object detector using raw and unlabeled videos and extracted speech

An example system includes a processor to receive input data comprising noisy positive data and clean negative data. The processor is to cluster the input data. The processor is to compute a potential score for each cluster of the clustered input data. The processor is to iteratively refine cluster quality of the clusters using the potential scores of the clusters as weights. The processor is to train a classifier by sampling the negative dataset uniformly and the positive set in a non-uniform manner based on the potential score.

Classifier training using noisy samples

An example system includes a processor to receive a number of video frames. The processor is to apply a grid to each video frame of the number of video frames and generate features for each cell in the grid. The processor is to calculate distances between matching regions of two consecutive frames. The processor is to apply max pooling followed by min pooling in horizontal regions of the number of video frames. The processor is to apply max pooling followed by min pooling in vertical regions of the number of video frames. The processor is to detect an application switch in response to detecting both a horizontal change and a vertical change between the two consecutive video frames exceed a threshold.

Growing community of inventors

Zikhron Yaakov, Israel

Elad Amrani