Facilitating model output modifications via physical gesture directed to portion of generative output

Abstract

Implementations set forth herein relate to modifying a generative output of an application according to an input gesture that is performed without necessarily interacting with a GUI element that is rendered separate from a generative output (e.g., a GUI element separate from a natural language output generated using an LLM or an image generated using an image diffusion model). Various different input gestures can be performed by a user to refine a generative output to be simpler, more complex, to include an image, to modify a generated image, and/or otherwise modify the generative output. In some implementations, an input gesture can be processed as one or more predetermined gestures, and/or an input gesture can be interpreted per case using an available model for interpreting such gestures. In this way, models for interpreting gestures and/or refining generative output can be enhanced through further training of such models.