July 26, 2010 posted by Jean-Yves Migeon
With kernel revision 5.99.37, the options(4) PAE was added to native i386.
It is currently disabled by default.
PAE, or Physical Address Extension, is a mode that started to appear with
Intel's Pentium Pro processor. When enabled, the i386 memory management physical addresses, including page directory and page table
entries, are promoted to 64 bits entities, instead of 32 bits. This allows,
in the present state, to address physical accesses with 36 bits -- thus turning the
whole physical address space to 64GB (although the userland virtual address
space remains with 32 bits addresses, or 4GB).
As NetBSD supported amd64 very early, there was no real urge to add PAE
support within the kernel; in early 2002, hosts with more than 4GB were
rare, and those that had more than 4GB of memory were already moving to amd64.
Historically, the first appearance of PAE was thanks to Manuel Bouyer
(bouyer@), for the Xen port. It remains, even today, the only solution to run
32 bits domUs with a 64 bits Xen hypervisor. The situation became even more
strict starting with Xen 3.3, where non-PAE support was removed from Xen,
effectively forcing the domains (dom0 as well as domUs) to move to full PAE
Later, Jeremy Morse took interest in having PAE supported within native i386,
and proposed a patch on port-i386@ for it. I took the responsibility for
merging it within -current, and make it less intrusive with regards to the
present code of port-xen.
In essence, adding PAE within NetBSD was not a difficult task; however, it
took quite a lot of time for testing and debugging, as the merge with the
current required modifications in low level code (boot and initialization,
handling), as well as fixes in place where physical address change
could mask the upper 32 bits (addresses could not be considered as 32 bits "unsigned long" anymore). Fortunately, the API in NetBSD being very
clear, finding out and isolating the problematic parts was easy. Besides,
having PAE inside GENERIC forced the implementation to be multi-processor safe,
so the Xen port can later take advantage from it and move more easily to the
multi processor world.
For those interested in small security improvements, enabling PAE on i386
has the benefit of unmasking the 63rd bit in the physical address, called the
NX/XD (No-eXecute/eXecute Disable) bit. By marking a physical page with this
bit, you can prevent code execution on the page. All CPUs do not support this
feature; you can easily spot it through cpuctl(8) -- look for NOX or XD in
the features output.
Importing PAE was an interesting challenge, as it raised concerns regarding
the stability of the kernel ABI when manipulating physical addresses.
Physical addresses are constantly used for device drivers, as they are needed
for communication with them over different types of buses. Stabilizing the
ABI offers the possibility to develop drivers, or modules, without fear of
breaking binary interfaces. Here, it will help modularizing the kernel even
further, by providing modules, and hopefully, a kernel, that could
fit native, PAE and Xen memory models without needing separate compile and
build time options.