Hands on experience with EdgeRouter ERLite-3

After spending last year on improving ARM test coverage (and then fixing bugs when found), which greatly improved ARM test results, I now turned to MIPS. I used to have some SGI O2 machines, but for some (yet unclear) reason, none of them powers up any more. I also had an alchemy base MeshCube, but that broke too.

Some time ago, Ingenics donated a CI20 board to me and I am in the process of getting that testable. Michael Lorenz did most of the needed work already, but there are issues with the network driver, which make automatic test runs hard.

When a few days ago cavium/octeon support was added to NetBSD-current, I just had to buy an ERLite-3 device:

This is probably the cheapest MIPS64 hardware currently available. It is in-store at many customer electronics shops all over the world.

The ERLite-3 uses a big-endian dual-core octeon chipset, has external serial console (via a cisco console cable) and three gigE ethernet ports. It also features an internal (built in) USB hard disk. It can be configured to boot the NetBSD kernel from that usb disk, or load it via tftp from another server. But booting from internal "disk" plus multiple gigE ports makes it an ideal device for a NetBSD based home or small office firewall, DNS cache, DHCP server setup.

From the NetBSD point of view this device is as plug&play as you can get. Connect the serial console cable, reboot it, press Ctrl-C during early boot and get to the u-boot prompt. Prepare a user land and kernel, like:

build.sh -m evbmips64-eb sets
build.sh -m evbmips64-eb kernel=ERLITE

The kernel build will produce a "netbsd" and "netbsd.elf32" file. Extract the created sets in the NFS root directory, cd to the dev directory there and run "sh MAKEDEV all". Put "netbsd" into the NFS root directory. Copy "netbsd.elf32" to the tftp server directory (I renamed mine to "erlite.elf32") and then go back to the edge router console.

dhcp
tftp $loadaddr erlite.elf32
bootoctlinux

and your kernel should boot. (Never mind the funny name of the boot command).

The obligatory dmesg:

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015
    The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 7.99.15 (ERLITE) #1: Wed May  6 21:40:41 CEST 2015
        martin@night-owl.duskware.de:/usr/src/sys/arch/evbmips/compile/ERLITE
total memory = 512 MB
avail memory = 490 MB
timecounter: Timecounters tick every 10.000 msec
mainbus0 (root)
cpu0 at mainbus0: 500.00MHz (hz cycles = 5000000, delay divisor = 500)
cpu0: Cavium CN50xx (0xd0601) Rev. 1 with software emulated floating point
cpu0: 64 TLB entries, 512TB (49-bit) VAs, 512TB (49-bit) PAs, 256MB max page size
cpu0: 32KB/128B 4-way set-associative L1 instruction cache
cpu0: 16KB/128B 64-way set-associative write-back coherent L1 data cache
iobus0 at mainbus0
iobus0: initializing POW
iobus0: initializing FPA
com0 at iobus0: address=0x0001180000000800: ns16650, no ERS, working fifo
com0: console
com at iobus0: address=0x0001180000000c00 not configured
octeon_rnm0 at iobus0: address=0x0001180040000000
octeon_rnm0: random number generator enabled: 1hz
octeon_twsi at iobus0: address=0x0001180000001000 not configured
octeon_mpi at iobus0: address=0x0001070000001000 not configured
octeon_gmx0 at iobus0: address=0x0001180008000000
cnmac0 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac0: Ethernet address 04:18:d6:f0:4b:e4
atphy0 at cnmac0 phy 7: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
cnmac1 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac1: Ethernet address 04:18:d6:f0:4b:e5
atphy1 at cnmac1 phy 6: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
cnmac2 at octeon_gmx0: address=0x0001180008000000: RGMII
cnmac2: Ethernet address 04:18:d6:f0:4b:e6
atphy2 at cnmac2 phy 5: Atheros AR8035 10/100/1000 PHY, rev. 2
atphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseSX-FDX, 1000baseT-FDX, auto
dwctwo0 at iobus0: address=0x0001180068000000
usb0 at dwctwo0: USB revision 2.0
bootbus0 at mainbus0
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "mips3_cp0_counter" frequency 500000000 Hz quality 100
uhub0 at usb0: vendor 0000 DWC2 root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
umass0 at uhub0 port 1 configuration 1 interface 0
umass0: vendor 13fe USB DISK 2.0, rev 2.00/1.00, addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, 1 lun per target
sd0 at scsibus0 target 0 lun 0: <, USB DISK 2.0, PMAP> disk removable
sd0: 3824 MB, 959 cyl, 255 head, 32 sec, 512 bytes/sect x 7831552 sectors
boot device: 
root device: cnmac0
dump device: 
file system (default generic): 
root on cnmac0
mountroot: trying nfs...
nfs_boot: trying DHCP/BOOTP
cnmac0: link state UP (was UNKNOWN)
cnmac0: link state DOWN (was UP)
cnmac0: link state UP (was DOWN)
nfs_boot: DHCP next-server: 192.168.150.188
nfs_boot: my_domain=duskware.de
nfs_boot: my_addr=192.168.150.192
nfs_boot: my_mask=255.255.254.0
nfs_boot: gateway=192.168.151.1
root on 192.168.150.188:/hosts/erlite
root time: 0x554ad8bc
root file system type: nfs
kern.module.path=/stand/evbmips/7.99.15/modules
WARNING: no TOD clock present
WARNING: using filesystem time
WARNING: CHECK AND RESET THE DATE!
init path (default /sbin/init): 
init: copying out path `/sbin/init' 11
pid 1(init): ABI set to N32 (e_flags=0x20000027)

The port to octeon (thanks IIJ!) is new, but already quite complete. The second cpu core is not yet started, but I hope this will be fixed very soon. No MIPS has been part of the automatic test runs recently, so there is some fallout, as expected. On first try I got several core files, among others from pkg_info, pkg_admin and gdb. The latter makes analyzing the crashes a bit more challenging, but give me a few days ;-}

[4 comments]