Control flow redirection and analysis for detecting vulnerability exploitation

Abstract

A security module detects attempted exploitations of vulnerabilities of applications executing on a computer. The security module hooks an application on the computer. The hook transfers control flow to the security module if execution reaches a hooked location. When a hook is followed, the security module saves the state of the computer and activates an analysis environment. A virtual machine within the analysis environment executes signatures that programmatically analyze the state of the computer to determine whether a vulnerability in the application is being exploited. If a signature detects an exploit, the security module blocks the exploit by skipping over the one or more instructions that constitute the exploit, terminating the application, or performing a different action. The security module reports the detected exploit attempt to the user of the client. The security module returns control flow back to the application if it does not detect an exploit.