System and method for preventing exploitation of large language

Abstract

A method for preventing exploitation of large language models is disclosed. The method includes receiving and analyzing API calls, including user requests and LLM-generated responses, to detect vulnerabilities. The method uses an encoder-based model to flag suspicious API calls based on contextual analysis, assigns toxicity scores to responses for harmful content, evaluates plugin vulnerabilities for malware risks, and detects private data leaks using trained detection models. Then, a vulnerability detection model integrates flagged API calls, toxicity scores, assessed vulnerabilities, and detected data leaks to generate a comprehensive risk assessment. Based on this assessment, the method performs mitigation actions, including blocking flagged API calls or harmful responses, restricting access to sensitive data, and alerting users to potential leaks.