Unbloating the VAX install CD

A recent discussion on the port-vax mailing list brought a problem with the default installation method (when booting from CD, which typically is the easiest way) to my attention: it would not work on machines with 16 MB RAM or less.

So, can we do better? Looking at the size of a GENERIC kernel:

   text    data     bss     dec     hex filename
2997389   67748  173044 3238181  316925 netbsd

it seems we can not easily go below 4 MB (and for other reasons we would need to compile the bootloader differently for that anyway). But 16MB is still quite a difference, so it should work.

Now at the time I started this quest, I only had one VAX machine in real hardware - a VaxStation 4000 M96a, one of the fastest machines, and with 128 MB RAM well equipped. This is nice if you try to natively compile modern gcc, but I did not feel like fiddling with my hardware to create a better test environment for small RAM installations.

Like a year (or so) ago, when I fixed the VAX primary boot blocks (with lots of help from various vaxperts on the port-vax mailing list), SIMH, found in pkgsrc as emulators/simh, proved helpful. Testing various configurations I found an emulated VAX 11/780 with 8 MB to be the smallest I could get working.

The first step of the tuning was obvious: the CD image used a ramdisk based kernel, with the ramdisk containing all of the install system. At the same time, most of the CD was unused. We already use different schemes on i386, amd64 and sparc64 - so I cloned the sparc64 one and adjusted it to VAX. Now we use the GENERIC kernel on CD and mount the ISO9660 filesystem on the CD itself as root file system. The VAX boot loader already could deal with this, only a minor fix was needed for the kernel to recognize some variants of CD drives as boot device.

The resulting CD did boot, but did not go far in userland. The CD did only contain a (mostly) empty /dev directory (without /dev/console), which causes init(8) to mount a tmpfs on /dev and run the MAKEDEV script there. But to my surprise, on the 11/780 mfs was used instead of tmpfs - and we will see why soon. Next step in preparation of the userland for the installer is creating additional tmpfs instances to deal with the read-only nature of the CD used as root. This did not work at all, the mount attempts simply failed - and the installer was very unhappy, as it could not create files in /tmp for example.

I checked, but tmpfs was part of the VAX GENERIC kernel. I tried the install CD on a simulated MicroVAX 3900 with 64 MB of RAM - and to my surprise all of /dev and the three additional tmpfs instances created later worked (as well as the installation procedure). I checked the source (stupid me) and then found the documentation: tmpfs reserved a hard coded 4 MB of RAM for the system. With the GENERIC kernel booted on a 8 MB machine, we had slightly less than 4 MB RAM free, so tmpfs never worked.

One step back - this explained why /dev ended up as a mfs instead of tmpfs. The MAKEDEV code is written to deal with kernels that do include tmpfs, but also with those that do not: it tried tmpfs, and falls back to mfs if that does not work. This made me think, I could do the same (but without even trying tmpfs): I changed the install CD scripts to use mfs instead of tmpfs. The main difference is: mfs uses a userland process to manage the swappable memory. However, we do not have any swap space yet. Checking when sysinst enables swapping for the first time, I found: it never did on VAX. Duh! I added the missing calls to machine dependent code in sysinst, but of course the installer can only enable swap after partitioning is done (and a swap partition got created).

Testing showed: we did not get far enough with four mfs instances. So let us try with fewer. One we do not need is the /dev one: I changed the CD content creation code to pre-populate /dev on the CD. This is not possible with all filesystems, including the original ISO9660 one, but with the so-called Rockridge Extensions it works. We know that it is a modern NetBSD kernel mounting the CD - so support for those extensions is always present. I made some errors and hit some bugs (that got fixed) on the way there, but soon the CD booted without creating a mfs (nor tmpfs) for /dev.

Still, three mfs instances did not survive until sysinst enabled swapping. The userland part was killed once the kernel ran out of memory. I needed tmpfs working with less than 4 MB memory free. After a slight detour and some discussion on the tech-kern mailing list, I changed tmpfs to deal (and only reserve a dynamically scaled amount of memory calculated bv the UVM memory management). With this change, a current install CD just works, and installation completes successful.

The following is just the start of the installation process, the sysinst part afterwards is standard stuff and left out for brevity.

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014                 
    The NetBSD Foundation, Inc.  All rights reserved.   
Copyright (c) 1982, 1986, 1989, 1991, 1993           
    The Regents of the University of California.  All rights reserved.
                                                                      
NetBSD 6.99.43 (GENERIC) #1: Thu Jun  5 22:01:14 CEST 2014
        martin@night-owl.duskware.de:/usr/obj/vax/usr/src/sys/arch/vax/compile/GENERIC
VAX 11/780
total memory = 8188 KB
avail memory = 3884 KB
mainbus0 (root)       
cpu0 at mainbus0: KA80, S/N 1234(0), hardware ECO level 7(112)
cpu0: 4KB L1 cachen, no FPA                                   
sbi0 at mainbus0           
mem0 at sbi0 tr1: standard
mem1 at sbi0 tr2: standard
uba1 at sbi0 tr3: DW780   
dz1 at uba1 csr 160100 vec 304 ipl 15
mtc0 at uba1 csr 174500 vec 774 ipl 15
mscpbus0 at mtc0: version 5 model 5   
mscpbus0: DMA burst size set to 4  
uda0 at uba1 csr 172150 vec 770 ipl 15
mscpbus1 at uda0: version 3 model 6   
mscpbus1: DMA burst size set to 4  
de0 at uba1 csr 174510 vec 120 ipl 15: delua, hardware address 08:00:2b:cc:dd:ee
mt0 at mscpbus0 drive 0: TU81                                                  
mt1 at mscpbus0 drive 1: TU81
mt2 at mscpbus0 drive 2: TU81
mt3 at mscpbus0 drive 3: TU81
ra0 at mscpbus1 drive 0: RA92
ra1 at mscpbus1 drive 1: RA92
racd0 at mscpbus1 drive 3: RRD40
ra0: size 2940951 sectors       
ra1: no disk label: size 2940951 sectors
racd0: size 1331200 sectors             
boot device: racd0         
root on racd0a dumps on racd0b
root file system type: cd9660 
init: kernel secur           

You are using a serial console, we do not know your terminal emulation.
Please select one, typical values are:

        vt100
        ansi
        xterm

Terminal type (just hit ENTER for 'vt220'): xterm
                                                                                
 NetBSD/vax 6.99.43                                                             
                                                                                
 This menu-driven tool is designed to help you install NetBSD to a hard disk,   
 or upgrade an existing NetBSD system, with a minimum of work.                  
 In the following menus type the reference letter (a, b, c, ...) to select an   
 item, or type CTRL+N/CTRL+P to select the next/previous item.                  
 The arrow keys and Page-up/Page-down may also work.                            
 Activate the current selection from the menu by typing the enter key.          


                +---------------------------------------------+
                |>a: Installation messages in English         |
                | b: Installation auf Deutsch                 |
                | c: Mensajes de instalacion en castellano    |
                | d: Messages d'installation en français      |
                | e: Komunikaty instalacyjne w jezyku polskim |
                +---------------------------------------------+

Overall this improved NetBSD to better deal with small memory systems. The VAX specific install changes can be brought over to other ports as well, but sometimes changes to the bootloader will be needed. [2 comments]