Making RockPro64 a NetBSD Server

The time has come to upgrade my SunBlade 2500s to something more power friendly and faster. I'd already removed one CPU and thus half the ram from two of these systems to reduce their power consumption, but it's still much higher than it could be.

After much searching, I've decided on Pine64's RockPro64 4GiB ram model (technically, only 3.875GiB ram.) Pine64 make SBCs, laptops, phones, and various other mostly ARM gadgets, and the RockPro64 has the fastest CPU they ship (Rockchip RK3399), and can use a small "NAS Case", that is sufficient to house 2 HDDs and, at a stretch, upto 6 SSDs (cooling would become quite an issue at this point.)

In my SATA setup, I have 3 SSDs with a JMicron 585 card in the PCIe slot, two SSDs in the NAS case SSD region, and the third is in the HDD region with an adapter. I have used two SATA power splitters to convert the NAS case's 2 SATA power ports into 4, with the 4th one also powering a Noctua case fan. The cabling is not great with this, with enough SATA power cabling for 6 devices to lay. Probably a bespoke power cable to connect to the RockPro64 would make this nicer and probably improve cooling slightly, but I'm just using off-the-shelf components for now.

In the last year or so I've been working on making NetBSD/arm64 big-endian more featureful. In particular, I've added support for:

• other-endian access disklabels, FFS file-systems in the NetBSD libsa
• the "look 64 sectors later" for RAIDFrame partitions in MI efiboot (the x86 specific efiboot has more extensive support for this that should be ported over)
• other-endian access to RAIDFrame labels in the kernel
• updated the U-Boot package and featureset to include AHCI/SATA support, workaround some bugs and fix the newer U-Boot SPI loader location, and figured out how to default loading from either SATA or NVMe

There are not too many special actions needed for this sort of setup compared to a normal NetBSD or Arm system. While I built my installations by hand, the standard NetBSD Arm images are suitable for this task. It's easiest to start from an SD card with the RockPro64 u-boot installed. There are two U-Boot images available, one for SD/eMMC, and one for SPI (there is an odd problem with the early SPI loader that requires a portion of the image to be different.) The pkgsrc package for sysutils/u-boot-rockpro64 version 2022.01 has these suggested methods for installing the U-Boot image (this package should be buildable on any supported pkgsrc platform).

To install U-Boot into the SD/eMMC card (can run on any system, the image must be written at 32KiB into the device):

# dd if=/usr/pkg/share/u-boot/rockpro64/rksd_loader.img seek=64 of=/dev/rld0c

To install U-Boot into the SPI flash (must be run on the host, and lives at the very start of the device:

# dd if=/usr/pkg/share/u-boot/rockpro64/rkspi_loader.img bs=64k of=/dev/spiflash0

When booting from NVMe or SATA, one must drop to the U-Boot prompt and adjust the "boot_targets" value to put scsi* (SATA) or nvme* ahead of the mmc* and usb* options.

The original value for me:

=> printenv boot_targets
boot_targets=mmc1 usb0 mmc0 nvme0 scsi0 pxe dhcp sf0

Which is then adjusted and saved with eg:

=> setenv boot_targets nvme0 scsi0 mmc1 usb0 mmc0 pxe dhcp sf0
=> saveenv
Saving Environment to SPIFlash... Erasing SPI flash...Writing to SPI flash...done
OK

(In this list, mmc1 is the SD slot, mmc0 is the eMMC card, pxe and dhcp are netbooting, and sf0 attempts to load further U-Boot scripts from SPI.)

There are some minor issues with the RockPro64. It has no ability to use ECC memory. It only comes with 1 PCIe 4x slot, and Rockchip errata limited this to PCIe 1.x (though no NetBSD users encounted any issues with PCIe 2.x enabled, and this can be forced back on via a DTS patch.) It is possible to use a PCIe bridge (I have never done this, though I would like to try in the future) to enable more devices for booting, or to enable both a network and storage device.