HugePages (Linux) and Large Pages (Windows) references
With the arrival of “cheap” Intel/AMD based servers with 64-bit CPU’s and large amount of memory, I think the DBA managing Oracle on Linux/Windows has to be aware of kernel features that allows better handling of otherwise, vast amount of standard memory pages (4k on Linux and Windows). Lately I was doing some research of the available articles on Metalink about Linux and Windows approaches to handle larger than default (4k) memory pages.
While finding documentation about HugePages feature on Linux was easy, the lack of the same information for Windows equivalent, Large Pages mechanism, was somehow a frustrating experience – I hope Oracle will refresh some of the existing notes that covers NT architecture with the information about Large Pages. The concept behind the HugePages & Large Page are very similar, but as we know, usually the devil is hiding in the details.
Below are references that I used during my research.
Linux – HugePages
Metalink Notes:
- Note:361323.1 “HugePages on Linux: What It Is… and What It Is Not…”
- Note:361468.1 “HugePages on 64-bit Linux”
- Note:401749.1 “Shell Script to Calculate Values Recommended HugePages / HugeTLB Configuration”
- Note:275318.1 “The Bigpages Feature on Linux”
- Note:261889.1 “Bigpages vs. Hugetlb on RedHat Linux”
- Note:46001.1 “Oracle Database and the Windows NT memory architecture, Technical Bulletin”
- Note:46053.1 “Windows NT Memory Architecture Overview”
- MS Windows Internals, by Mark E. Russinovich and David Salomon, Fourth Edition
- MSDN- Large Page Support (targeted at developers)
- The Memory Manager in Windows Server 2003 and Windows Vista (ppt)
- AMD: Supersizing Java: Large Pages on the Opteron Processor, Part 1
- AMD: Supersizing Java: Large Pages on the Opteron Processor, Part 2
- VMware: Large Page Performance
- Which Windows versions has support for Large Pages? I’m not 100% sure I know the answer. According to Wikipedia the Large Pages support is available on Windows 2003 SP1, but not on Windows XP/Vista. I don’t think this is entirely correct, since Windows XP is mentioned in Windows Internals book on page 383 (see reference above)
- don’t use Oracle LOCK_SGA in conjunction with Large Page, instead refer to PRE_PAGE_SGA parameter, if you wish to pre-allocate large pages in memory at instance startup
- user account under which process that wants to use Large Pages is running, needs “Lock Pages in Memory” system privilege. Oracle process by default runs under System account
which already has this privilege…[this is false, by default no user account has this privilege, hence adding “System” account (or dedicated user account, if used!) is mandatory] - not running Oracle service under dedicated user (such as local user oracle) means that service will run under account SYSTEM. If we authorize SYSTEM account to use “Lock Pages in Memory” privilege, we’re giving this privilege to several other services. This brings some security/stability concerns to the table. Shouldn’t we mandatory run Oracle service under some user account, rather than SYSTEM, if we’re going to use Large Pages?
- memory pages allocated via Large Page are pinned in memory and doesn’t swap to the pagefile, make sure you’re not allocating too much memory (for example, over allocating SGA) and thus taking away the memory from the regular VM space that is needed for client sessions (Oracle shadow “processes”)
- each Large Page must consist of contiguous memory, so for example if the size of the page is extremely large it can happen that due to the memory fragmentation, the process can not allocate all requested large pages (the potential for the problem rise with the number of different running processes and the frequency of starting/stopping the process that is using Large Pages)
- once allocated, the memory pinned as Large Page is not released until the process stops
- Windows settings that control Large Page behavior are stored under registry key:
HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory - according to “Windows Internals” book, the default large page sizes depends on architecture used:
x86 = 4MB (2MB if PAE is enabled)
x64 = 2MB
IA64 = 16MB
Bigpages Linux kernel feature was replaced by HugePages in 2.6 backported to 2.4), for those of you who are “stuck” on older kernel version (such as RHAS 2.1), two Metalink notes:
Other references:
Windows Server 2003 (x86, EMT64)
Oracle introduced support for Large Pages in Windows 2003 (SP1) in Oracle 10g R1.
I would expect to find the necessary information in the following two Metalink notes:
…unfortunately not.
The Chapter 7, Tuning Windows to Optimize Oracle Database in Oracle Database Platform Guide 10g Release 2 (10.2) for Microsoft Windows (x64) is much more helpful. This is all you need to setup Oracle 10g/11g to use Large Pages on Windows 2003.
Microsoft official documentation:
Other sources:
My personal side notes about “Large Pages” on Windows 2003:
Posted on 06.01.2008, in Oracle and tagged oracle. Bookmark the permalink. 2 Comments.
Large Pages & Oracle10g – a feedback from the trench
Sorry folks, I think I owe this comment for more than a month….as you might know, we failed to use Large pages on Windows 2003 x64 (SP2). We don’t know what exactly we’re doing wrong, but we simply could not make Oracle 10.2.0.3 instance to allocate large memory pages for SGA. Instructions available in Chapter 7 (see link above) were followed and checked word by word. The setup instructions itself are trivial – that’s why the “pain” is stronger :-).
As you know it’s about all or nothing deal when it comes to Large pages – Oracle will allocate all pages as Large Pages or none. At the beginning we thought memory fragmentation is the root cause for the problem, but even after a fresh reboot and trying with much smaller SGA sizes compared to available physical memory, Oracle never allocated memory with large pages. During one experiment we even managed to bring the machine to become unresponsive.
Test Case 1)
On machine with 16GB of RAM we tried to allocate 8GB SGA with Large Pages.
The end result was that the instance started, but eventually the pages allocated to Oracle.exe process started to being paged out of memory. This should not happen if memory pages were indeed allocated via “Large Pages” mechanism, since those are non page-able, if our understanding is correct. Our speculation is, that Oracle tried initially to allocate SGA with Large Pages, and due to either OS bug or memory fragmentation, simply gave up and allocated SGA with ordinary, 4K pages.
Test Case 2)
On machine with 32GB of memory we tried to allocate 16GB SGA with Large Pages – and this experiment was almost “deadly” for the machine.
Shortly after we issued “startup” command for Oracle instance, the machine became practically unresponsive.
The only way we could cleanly shutdown the server was by issuing remote shutdown (it took close to 30 minutes for the command to kick in – some sort of CPU race condition was obviously going on). Fortunately, we tested the configuration with Oracle service being setup to start in “Manual” mode, so we could fix the problem by removing ORA_%SID%_LPENABLE=1 parameter from registry key.
For now, we abandoned the idea to use Large Pages on Windows 2003 x64. If anyone has a suggestion or want to share first hand experience about this feature with us…we are all ears.
Large pages (apparently) doesn’t work with 10g on Windows 2003
Let me start this comment with one additional Metalink note that can be useful for setting up large pages support:
422844.1 “Using Large Memory Pages on 64-Bit Windows Systems”
And now to the subject of this comment. It was a relief to read Howard J. Rogers blog post that firmly confirms my own experience with Oracle10g R2 and Windows 2003 (SP2). Large pages doesn’t work with Oracle10g on this Windows release. Howard got a recommendation to upgrade OS to Windows 2008. Apparently Oracle believes it’s Microsoft to blame for this bug. I have a mixed feelings about that Oracle statement, for now. It would be nice if Oracle could address this issue (bug) with dedicated Metalink note, sparing us to waste our time.