Code generation system using pre-trained diffusion model

Abstract

A code generating system is configured to receive a code description prompt which includes a natural language description of a code snippet for the code generating system to generate via a user interface. The system encodes the prompt to generate a code description embedding that embeds the natural language prompt into a continuous space using an encoder. A noisy code snippet embedding is then iteratively denoised conditioned on the code description embedding to generate a denoised code snippet embedding. The denoised code snippet embedding is decoded using a pre-trained decoder having full self-attention and cross-attention with the code snippet embedding to generate a final predicted code snippet embedding. The final predicted code snippet embedding is then projected to discrete code tokens using a classification head that computes a probability distribution over code tokens and selects code tokens for a predicted code snippet based on the probability distribution.