Orchestrating dynamic actions through multi-modality inputs

Abstract

Certain aspects of the disclosure provide for processing multi-modality inputs to trigger dynamic actions. In examples, a method may include: receiving user input through one or more input modalities; converting the user input and one or more supported actions obtained from an actions repository into a unified schema; applying a set of moderation rules to the unified schema to generate a moderated input; generating a prompt based on the moderated input and one or more influencing examples; processing the prompt using one or more Large Language Models (LLMs) to obtain one or more matched actions and populated data models; augmenting the one or more matched actions with additional data; initiating one or more supported actions based on the one or more matched actions and populated data models; and generating a response to the user input based on results of executing the one or more supported actions.