August 08, 2014 posted by Antti Kantee
The most time-consuming part of operating system development is obtaining
enough drivers to enable the OS to run real
applications which interact with the real world. NetBSD's rump kernels allow reducing
that time to almost zero, for example for developing special-purpose operating
systems for the cloud and embedded IoT devices. This article describes
an experiment in creating an OS by using a rump kernel for drivers.
It attempts to avoid going into full detail on the principles
of rump kernels,
which are available for interested readers from
rumpkernel.org. We start by defining
the terms in the title:
- OS: operating system, i.e. the overhead that enables applications to run
- internet-ready: supports POSIX applications and talks TCP/IP
- a week: 7 days, in this case the period between Wednesday night
last week and Wednesday night this week
- from scratch: began by writing the assembly instructions for the kernel entry point
- rump kernel: partial kernel consisting of unmodified NetBSD kernel
- bare metal: what you get from the BIOS/firmware
Why would anyone want to write a new OS? If you look at our definition
of "OS", you notice that you want to keep the OS as small as possible.
Sometimes you might not care, e.g. in case of a desktop PC, but other
times when hardware resources are limited or you have high enough security
concerns, you actually might care. For example, NetBSD itself is not able to
run on systems without a MMU, but the OS described in this article does
not use virtual memory at all, and yet it can run most of the same
applications as NetBSD can. Another example: if you want to finetune
the OS to suit your application, it's easier to tune a simple OS than a
very complicated general purpose OS. The motivation for this work came
in fact from someone who was looking to provision applications as services
on top of VMWare, but found that no existing solution supported the
system interfaces his applications needed without dragging an entire
classic OS along for the ride.
Let's move on to discussing what an OS needs to support for it to be able
to host for example a web server written for a regular OS such as
Linux or the BSDs. The list gets quite long. You need a file system
where the web server reads the served pages from, you need a TCP/IP
stack to communicate with the clients, and you need a network interface
driver to be able to send and receive packets. Furthermore, you need the
often overlooked, yet very surprisingly complicated system call handlers.
For example, opening a socket is not really very complicated to handle.
Neither is reading and writing data. However, when you start piling
things like fcntl(O_NONBLOCK) and poll() on top,
things get trickier. By a rough estimate, if you run an httpd on
NetBSD, approximately 100k lines of code from kernel are used just to
service the requests that the httpd makes. If you do the math (and
bc did), there are 86400 seconds in a week. The OS we are
discussing is able to run an off-the-shelf httpd, but definitely I did not
write >1 line of code per second 24/7 during the past week.
Smoke and Mirrors, CGI Edition
The key to happiness is not to write 100k lines of code from scratch, nor
to port it from another OS, as both are time-consuming and error-prone
techniques, and error-proneness leads to even more consumption of time.
Rump kernels come into the picture as the key to happiness and provide
the necessary drivers.
As the old saying goes: "rump kernels do
not an OS make", and we need the rest of the bits that make up the OS
side of the software stack from somewhere. These bits need to make it
seem like the drivers in a rump kernel are running inside the NetBSD
kernel, hence "smoke and mirrors". What is surprising is how little code
needs to exist between the drivers and the hardware, just some hundreds
of lines of code. More specifically, in the bare metal scenario we need
- low level machine dependent code
- thread support and a scheduler
- rump kernel hypercall layer
- additionally: bundling the application into a bootable image
The figure below illustrates the rump kernel software stack.
The arrows correspond to the above list (in reverse order).
We go over the list starting from the top of the list (bottom
of the figure).
Low level machine dependent
code is what the OS uses to get the CPU and devices to talking terms with
the rest of OS. Before we can do anything useful, we need to bootstrap.
Bootstrapping x86-32 is less work than one would expect, which incidentally
is also why the OS runs only in 32bit mode (adding 64bit support would not
likely be many hours of work — and patches are welcome). Thanks to the Multiboot
specification, the bootstrap code is more or less just a question
of setting the stack pointer and jumping to C code. In C code
we need to parse the amount of physical memory available and initialize
the console. Since NetBSD device drivers mainly use interrupts, we also need
interrupt support for the drivers to function correctly. On x86,
interrupt support means setting up the CPU's interrupt descriptor
tables and programming the interrupt controller. Since rump kernels
do not support interrupts, in addition we need a small interrupt
stub that transfers the interrupt request to a thread context which
calls the rump kernel. In total,
the machine dependent code is only a few hundred lines. The OSDev.org wiki contains a lot of
information which was useful when hammering the hardware into shape.
The other source of x86 hardware knowledge was x86 support in NetBSD.
Threads and scheduling might sound intimidating, but they are
not. First, rump kernels can run on top of any kinds
of threads you throw at them, so we can just use the ones which are
the simplest to implement: cooperative threads. Note, simple does
not mean poorly performing threads, and in fact the predictability of
cooperative threads, at least in my opinion, makes them
more likely to perform better than preemptive threading in
cases where you are honing an OS for a single application.
Second, I already had access to an implementation
which served as the basis: Justin Cormack's work on
fibers, which in turn has its roots in Xen MiniOS we use for
kernel on the Xen hypervisor, could be re-purposed as the threads+scheduler
implementation, with the context switch code kindly borrowed from MiniOS.
The rump kernel hypercall interface is what rump kernels themselves
run on. While the implementation is platform-specific, our baremetal OS
shares a large portion of its qualities with the Xen platform that was
already supported. Therefore, most of the Xen implementation applied
more or less directly. One notable exception to the similarities is
that Xen paravirtualized devices are not available on bare metal and
therefore we access all I/O devices via the PCI bus.
All we need now is the
application, a.k.a. "userspace". Support for application interfaces
(POSIX syscalls, libc, etc.) readily exists for rump kernels, so we
just use what is already available. The only remaining issue is building
the bundle that we bootstrap. For that, we can repurpose Ian Jackson's
app-tools which were originally written for the rump kernel Xen
platform. Using app-tools, we could build a bootable image containing
thttpd simply by running the app-tools wrappers for ./configure
and make. The image below illustrates part of the build output,
along with booting the image in QEMU and testing that the httpd really
works. The use of QEMU, i.e. software-emulated bare metal, is due to
You probably noticed that whole thing is just bolting a lot of working
components together while writing minimal amounts of necessary glue.
That is exactly the point: never write or port or hack what you can reuse
without modification. Code reusability has always been the strength of
NetBSD and rump kernels add another dimension to that quality.
The source code for the OS discussed in this post is
available under a 2-clause BSD license from repo.rumpkernel.org/rumpuser-baremetal.