System and method for enforcing application security policies using authenticated system calls

Abstract

Disclosed is an approach to system call monitoring in which authenticated system calls from an application are easily verified by an operating system kernel. The authenticated system call may be a system call augmented with extra arguments, which specify the policy for that call as well as a cryptographic message authentication code (MAC) that guarantees the integrity of the policy and the system call arguments. This extra information is used by the operating system kernel to verify the system call with little processing overhead. Versions of the applications in which regular system calls have been replaced by authenticated calls are generated automatically by a trusted installer program that reads the application binary, uses static analysis to generate policies, and then rewrites the binary with the authenticated calls. As a result, hacker attacks, malicious software and the like are less likely to be successful in compromising any computers or networks that employ such authenticated system calls.