Windows DEP – Data Execution Prevention

I think I’ll need to refer to Microsoft article “A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003” again in the future, that’s why I decided to write a short memo about DEP.

A week ago I received a Windows application compiled with Delphi. I tried to run application on our brand new server (Windows 2003 Standard Ed. R2, SP2, 32-bit, Intel Core Duo, 2.4GHz) with no success. I got “Address Violation at …[some hex address]” as soon as I tried to run the executable. The same application run successfully on Windows 2003 Standard Ed. R2, SP1, 32-bit (Pentium IV, 2GHz). The only difference is in CPU type and version of Service Pack.
Based on above article I assumed that software DEP is present in both versions of Windows 2003 – SP1 and/or SP2. So, I originally dismissed DEP as a potential cause for the error because it was turned on on both servers. This was a mistake.

The application provider suggested to turn off DEP feature completely on our new server and see if this helps. Instead, I decided to put Delphi application on the exception list for DEP and it helped. I believe it’s safer (but can also become more tedious) to list exceptions to the rule – especially if the same application is stored and run from different directories in which case you’ll have to browse to add the same application multiply times.

DEP

Conclusion: I believe SP2 for Windows 2003 server somehow changes the way, DEP actually works out of the box or it’s the (newer) CPU that changed the way OS is executing DEP.

Another thing to be careful about is to check DEP exception list after target executable is in any way changed (for example by replacing exe with the new one will trigger OS to remove application from the exception list!).

A quick search on Google revealed that this is a known problem with applications developed with Delphi and the one that Borland will have to deal with (perhaps they already fixed a compiler!?).

Posted on 17.12.2007, in MS Windows and tagged . Bookmark the permalink. Comments Off on Windows DEP – Data Execution Prevention.

Comments are closed.