Fine-tuning diffusion-based generative neural networks using singular value decompositions for text-to-image generation

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for fine-tuning diffusion-based generative neural networks in compact parameter spaces for text-to-image generation. In one aspect, a method performed by one or more computers for fine-tuning a diffusion-based generative neural network to obtain a fine-tuned version of the diffusion-based generative neural network is described. The method includes: for each of a number of neural network layers of the diffusion-based generative neural network: obtaining an initial weight matrix including a number of pre-trained weights parametrizing the neural network layer: performing a singular value decomposition on the initial weight matrix; and re-parametrizing the neural network layer with new weights that depend on spectral sifts; and training the spectral shifts of each of the number of neural network layers of the diffusion-based generative neural network to obtain the fine-tuned version of the diffusion-based generative neural network.