Multi-modal artifical neural network and a self-supervised learning method for training same

Abstract

A multi-modal artificial neural network and a self-supervised learning method for training that network. The learning method involves processing, using a first modality simple Siamese network, a pair of first modality augmented views of an input; processing, using a second modality simple Siamese network, a pair of second modality augmented views of the input; determining at least one cross-modal loss between the first and second modality simple Siamese networks; determining a total loss from: (i) first and second modality losses respectively determined during the processing using the first and second modality simple Siamese networks; and (ii) the at least one cross-modal loss; and training the first and second modality simple Siamese networks based on the total loss. The trained network may be used to analyze multi-modal content such as video content that has an audio track. A Multi-Modal Multi-Head Network (M3HN) may also be trained to process modality-specific and modality-agnostic representations.