PAE support for native i386

With kernel revision 5.99.37, the options(4) PAE was added to native i386. It is currently disabled by default.

PAE, or Physical Address Extension, is a mode that started to appear with Intel's Pentium Pro processor. When enabled, the i386 memory management physical addresses, including page directory and page table entries, are promoted to 64 bits entities, instead of 32 bits. This allows, in the present state, to address physical accesses with 36 bits -- thus turning the whole physical address space to 64GB (although the userland virtual address space remains with 32 bits addresses, or 4GB).

As NetBSD supported amd64 very early, there was no real urge to add PAE support within the kernel; in early 2002, hosts with more than 4GB were rare, and those that had more than 4GB of memory were already moving to amd64.

Historically, the first appearance of PAE was thanks to Manuel Bouyer (bouyer@), for the Xen port. It remains, even today, the only solution to run 32 bits domUs with a 64 bits Xen hypervisor. The situation became even more strict starting with Xen 3.3, where non-PAE support was removed from Xen, effectively forcing the domains (dom0 as well as domUs) to move to full PAE support.

Later, Jeremy Morse took interest in having PAE supported within native i386, and proposed a patch on port-i386@ for it. I took the responsibility for merging it within -current, and make it less intrusive with regards to the present code of port-xen.

In essence, adding PAE within NetBSD was not a difficult task; however, it took quite a lot of time for testing and debugging, as the merge with the current required modifications in low level code (boot and initialization, pmap(9) handling), as well as fixes in place where physical address change could mask the upper 32 bits (addresses could not be considered as 32 bits "unsigned long" anymore). Fortunately, the API in NetBSD being very clear, finding out and isolating the problematic parts was easy. Besides, having PAE inside GENERIC forced the implementation to be multi-processor safe, so the Xen port can later take advantage from it and move more easily to the multi processor world.

For those interested in small security improvements, enabling PAE on i386 has the benefit of unmasking the 63rd bit in the physical address, called the NX/XD (No-eXecute/eXecute Disable) bit. By marking a physical page with this bit, you can prevent code execution on the page. All CPUs do not support this feature; you can easily spot it through cpuctl(8) -- look for NOX or XD in the features output.

Importing PAE was an interesting challenge, as it raised concerns regarding the stability of the kernel ABI when manipulating physical addresses. Physical addresses are constantly used for device drivers, as they are needed for communication with them over different types of buses. Stabilizing the ABI offers the possibility to develop drivers, or modules, without fear of breaking binary interfaces. Here, it will help modularizing the kernel even further, by providing modules, and hopefully, a kernel, that could fit native, PAE and Xen memory models without needing separate compile and build time options.

[0 comments]