Supporting "high memory" on NetBSD/sgimips

March 10, 2009 posted by Jared McNeill

I purchased an SGI O2 some years back, and over time I have been finding little bits of time to improve support for the hardware on NetBSD. One of the things that has always bothered me is that NetBSD could not use more than 256MB of RAM, so I set out to fix it.

I typically spend most of my time in x86-land, so my knowledge of how MIPS works is weak at best. After speaking with a few people more familiar with the architecture, I learned that the memory size restriction is due to the way NetBSD's MIPS physical address map (pmap(9)) works. There is 512MB available for mapping physical addresses and some of that is typically used for devices. On the O2, the first 256MB is mapped to physical memory and the following 256MB is for memory mapped IO (for CRIME, MACE, PCI, etc).

On the O2, it appears that there are a few ways to access memory. A 256MB "low memory" window is mapped at 0x00000000, another 512MB "high memory" window at 0x20000000, and then a 1GB window at 0x40000000 that provides access to all available memory. Since the "high memory" window is at the 512MB mark, the kernel cannot directly access it. I decided to try to use the full memory window to allow use of all available memory.

As I did not feel comfortable modifying the MIPS pmap, I set out with my evil plan; I figured if I can access the memory at the 0x40000000 mark through a smaller mapping in kernel virtual address space, I should be able to expose this as a memory-backed block device. Using the md(4) driver as a reference, I added support to the crime(4) to expose a block device for any memory above the 256MB mark. Using the mips3_wired_enter_region() kernel function (from src/sys/arch/mips/mips/wired_mem.c), I am able to use 32MB of virtual address space in the block driver's strategy callback to copy in and out of high memory, reprogramming the TLB on demand.

Once this block device was in place, using the memory was as simple as creating a /dev/crime device node and using it as a swap device with higher priority than the swap partition on my hard disk.

With the changes, I see the following debug output at boot on my 448MB O2:

crime0 at mainbus0 addr 0x14000000: rev 1.1 (CRIME_ID: 161)
crime0: himem: 32 MB @ 0x50000000
crime0: himem: 32 MB @ 0x52000000
crime0: himem: 32 MB @ 0x54000000
crime0: himem: 32 MB @ 0x56000000
crime0: himem: 32 MB @ 0x58000000
crime0: himem: 32 MB @ 0x5a000000
crime0: himem: 192 MB memory disk

I then modified /etc/fstab to prioritize swap devices:

/dev/crime none swap sw,priority=0 0 0
/dev/sd0b none swap sw,priority=1 0 0

As a result I can now access the full amount of physical memory on my O2 with a 32-bit MIPS pmap!

A work in progress patch is available on the NetBSD FTP site.




G'd work but somehow misleading "high memory" term usage. As you have pointed out there is a serious design flaw in NetBSD/mips's pmap code which assumes all the physical memory accessible through KSEG1/KSEG0 address space. The design was (very) comfortable for two reasons; comfortable enough during the age when RAM was expensive and small (around 32-64MB range), and comfortable enough to handle physical memory contents without any help of MMU/TLB VPN/PFN mapping. However, there is no reason why the 32bit processor can not handle beyond 512MB physical memory. It should be just possible when pmap does the correct things for page_copy() and page_zero(). And the change will also solve 'VIPT' cache aliasing wreckage for the many post-R4000 age 32/64 processors whose 'effective cache size' exceeds 4KB, where term effective means distinctive virtual address range size for particular combination of whole size and cache way count.

Posted by Toru Nishimura on March 13, 2009 at 09:32 AM UTC #

All that low memory, low disk space, etc. gives me a headache. grrr...I'm no computer expert, but if someone knew an easier way to fix these errors, that would be stellar. Oh wow, operating all the physical memory! Kudos to you for figuring it out on your MIPS pmap!

Posted by Kavinase on November 15, 2010 at 08:12 PM UTC #

Hi there! Great work. I'm going to have a go at applying this to my tree. I'm really pleased some people are keeping the O2 alive. NetBSD 5.1 runs quite well. I have a few issues but I'm going to see if I can work through them. Again, thank you and please keep tinkering with sgimips :)

Posted by roop on January 09, 2011 at 08:04 PM UTC #

Post a Comment:
Comments are closed for this entry.