Generation of semantically modified variations of images with transformer networks

Abstract

A method for generating a semantically modified variation of an image. In the method: the image is divided into equally sized, non-overlapping patches; the patches are converted with a patch encoding function of a transformer network into a chain of tokens in a workspace; the tokens are grouped into preservation tokens, whose information is to be preserved in the variation, and masked tokens, whose information is to be masked in the variation; the preservation tokens are converted with an encoder of the transformer network into a chain of processed tokens; the chain is supplemented through application of an insertion operator that inserts these masked tokens at positions corresponding to the positions of the masked tokens in the original chain to form a chain that represents the sought variation; the chain is converted with a decoder of the transformer network into the sought variation.