Detection of indirect prompt injection attacks with malicious instructions detection models

Abstract

A malicious instructions detection model ('detector') intercepts augmented prompts destined for a large language model (“LLM”). Each augmented prompt was augmented with data from potentially compromised data sources susceptible to indirect prompt injection attacks. The detector tokenizes/preprocesses sentences in the augmented prompts and is invoked on the tokenized/preprocessed sentences to obtain confidence scores that each sentence comprises malicious instructions. If one or more of the confidence scores is above a threshold, the detector blocks the augmented prompt and generates an alert indicating the blocking and the malicious instructions. Otherwise, the detector communicates the augmented prompt to its intended LLM.