Channel fusion for vision-language representation learning

Abstract

Provided is an approach that aligns multi-modal tokens using cross-attention without losing the advantages of global self-attention. In contrast to previous works that concatenate the unimodal tokens along the sequence dimension, example approaches described herein align per-modality tokens by chaining them along the channels. Specifically, the tokens from one modality can be used to query the other modality and the output can be concatenated with the query tokens on the channels. An analogous process can also be repeated (or performed in parallel) where the roles of the two modalities are switched. The resulting sets of compound tokens can be concatenated and fed into a self-attention encoder such as a transformer encoder that performs self-attention.