Method and apparatus for verifying a suspect return pointer in a stack

Abstract

The present invention provides a computer implemented method, data processing system, and computer program product for verifying a return address. A computer stores the return address into a stack based on a function call. The computer generates a first hash based on a first stack frame and a second stack frame. The computer stores the first hash in a first canary location, wherein the first canary location is in the first stack frame. The computer executes at least one instruction of a routine referenced by the function call. The computer reads the first canary location to form a first suspect hash. The computer calculates a first verification hash based on the first stack frame and the second stack frame. The computer determines that the first verification hash matches the first suspect hash to form a first positive determination. The computer responsive to the first positive determination, the computer reads a second canary location to form a second suspect hash. The computer calculates a second verification hash based on the second stack frame. The computer determines that the second verification hash matches the second suspect hash to form a second positive determination. The computer responsive to the first positive determination and the second positive determination, the computer pops the return address off the stack. The computer executes at least one instruction at a memory location pointed to by the return address.