GSoC Reports: Fuzzing Rumpkernel Syscalls, Part 1

It has been a great opportunity to contribute to NetBSD as a part of Google Summer Of Code '20. The aim of the project I am working on is to setup a proper environment to fuzz the rumpkernel syscalls. This is the first report on the progress made so far.

Rumpkernels provide all the necessary components to run applications on baremetal without the necessity of an operating system. Simply put it is way to run kernel code in user space.

The main goal of rumpkernels in netbsd is to run,debug,examine and develop kernel drivers as easy as possible in the user space without having to run the entire kernel but run the exact same kernel code in userspace. This makes most of the components(drivers) easily portable to different environments.

Rump Kernels are constructed out of components, So the drivers are built as libraries and these libraries are linked to an interface(some application) that makes use of the libraries(drivers). So we need not build the entire monolithic kernel just the required parts of the kernel.

Why Honggfuzz?

I considered Honggfuzz the best place to start with for fuzzing the syscall layer as suggested by my mentors. LibFuzzer style library fuzzing method helped me in exploring how syscalls are implemented in the rumpkernel. With LibFuzzer we have the flexibility of modifying a few in-kernel functions as per the requirements to best suit the fuzzing target.

Fuzzing target

Taking a close look at src/sys/rump/librump/rumpkern/rump_syscalls.c we observe that this is where the rump syscalls are defined. These functions are responsible for creating the arguments structure (like wrappers) and passing it to syscalls which is define

rsys_syscall(num, data, dlen, retval)

which is defined from

rump_syscall(num, data, dlen, retval)

This function is the one that invokes the execution of the syscalls. So this should be the target for fuzzing syscalls.

Fuzzing Using Honggfuzz

Initially we used the classic LibFuzzer style.

int
LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size) 
{
    if(Size != 2 + 8+sizeof(uint8_t))
        return 0;

    ExecuteSyscallusingtheData();
    return 0;
}

However this approach into issues when we had to overload copyin(), copyout(), copyinstr(), copyoutstr() functions as the pointers that is used in these functions are from the Data buffer that the fuzzer provides for each fuzz iteration.

int
copyin(const void *uaddr, void *kaddr, size_t len)
{
..
..
..
..
     if (RUMP_LOCALPROC_P(curproc)) {
         memcpy(kaddr, uaddr, len); <- slow
     } else if (len) {
         error = rump_sysproxy_copyin(RUMP_SPVM2CTL(curproc->p_vmspace),
             uaddr, kaddr, len);
     }
     return error;
 }

Honggfuzz provides us HF_ITER interface which will be useful to actively fetch inputs from the fuzzer for example.

extern void HF_ITER(uint8_t **buf, size_t *len);

int
main()
{
  for (;;) {
    uint8_t *buf;
    size_t len;
    HF_ITER(&buf, &len);
    DoSomethingWithInput(buf, len);
  }
  return 0;
}

So we switched to a faster method of using HF_ITER() in honggfuzz to fetch the data from the fuzzer as it is relatively flexible to use.

EXTERN int
rumpns_copyin(const void *uaddr, void *kaddr, size_t len)
{
        int error = 0;
        if (len == 0)
                return 0;
        //HF_MEMGET() is a wrapper around HF_ITER()
        HF_MEMGET(kaddr, len);
        return error;
}

Similar overloading is done for copyout(), copyinstr(), copyoutstr().

The current efforts to fuzz the rump syscalls using honggfuzz can be found here.

This gave quite a speed bump to the "dumb" fuzzer from few tens iterations to couple of hundreds as we replaced the memcpys with a wrapper around HF_ITER().

Further work by Kamil Rytarowski

Kamil has detected that overloading copyin()/copyout() functions have a shortcoming that we are mangling internal functionality of the rump that uses this copying mechanism, especially in rump_init(), but also in other rump wrappers, e.g. opening a file with rump_sys_open().

The decision has been made to alter the fuzzing mechanism from pumping random honggfuzz assisted data into the rump-kernel APIs and intercept copyin()/copyount() family of functions to add prior knowledge of the arguments that are valid. The initial target set by Kamil was to reproduce the following rump kernel crash (first detected with syzkaller in the real kernel):

#include <sys/types.h>
#include <sys/ioctl.h>

#include <rump/rump.h>
#include <rump/rump_syscalls.h>

int
main(int argc, char **argv)
{
        int filedes[2];

        rump_init();
        rump_sys_pipe2(filedes, 0);
        rump_sys_dup2(filedes[1], filedes[0]);
        rump_sys_ioctl(filedes[1], FIONWRITE);
}

https://www.netbsd.org/~kamil/panic/rump_panic.c

We can compare that panicking the real kernel was analogous to the rump calls:

#include <sys/types.h>
#include <sys/ioctl.h>

#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <fcntl.h>


int
main(int argc, char **argv)
{
    int filedes[2];
    pipe2(filedes, 0);
    dup2(filedes[1], filedes[0]);
    ioctl(filedes[1], FIONWRITE);

    return 0;
}

https://www.netbsd.org/~kamil/panic/panic.c

Thus, Kamil wrote a new fuzzing program that uses pip2, dup2 and ioctl APIs exclusively and has prior knowledge about valid arguments to these functions (as the reproducer is using perfectly valid NetBSD syscalls and arguments):

https://github.com/adityavardhanpadala/rumpsyscallfuzz/blob/master/honggfuzz/ioctl/ioctl_fuzz2.c

The code instead of checking out impossible (non-existent) kernel code paths, it ensures that e.g. ioctl(2) operations are always existing ioctl(2) operations.

static unsigned long
get_ioctl_request(void)
{
    unsigned long u;

    // ioctlprint -l -f '\t\t%x /* %n */,\n'
    static const unsigned long vals[] = {
        0x8014447e /* DRVRESUMEDEV */,
        0xc0784802 /* HPCFBIO_GDSPCONF */,
        /* A lot of skipped lines here... */
        0xc0186b01 /* KFILTER_BYNAME */,
        0x80047476 /* TIOCSPGRP */,
    };

    u = get_ulong() % __arraycount(vals);

    return vals[u];
}

Kamil also rewrote the HF_MEMGET() function, so instead of recharging the buffer from the fuzzer, whenever the buffer expired the fuzzed program terminates resetting its state and checks another honggfuzz input. This intends to make the fuzzing process more predictable in terms of getting good reproducers. So far we were unable to generate good reproducers and we are still working on it.

Unfortunately (or fortunately) the code in ioctl_fuzz2.c triggers another (rump)kernel bug, unrelated to the reproducer in rump_panic.c. Furthermore, the crash looks like a race condition that breaks randomly, sometimes and honggfuzz doesn't generate good reproducers for it.

The obvious solution to this is to run the fuzzing process with TSan involved and catch such bugs quickly, unfortunately reaching MKSANITIZER for TSan + rumpkernel is still unfinished and beyond the opportunities during this GSoC.

Obstacles

• The fuzzer is still dumb meaning that we are still using just random data as arguments to the fuzzer so the coverage did not show that much improvement b/w 13 mins and 13 hours of fuzzing.
• Crash reproducers get sliced due to the way we are fetching input from fuzzer using HF_ITER() and as functions like copyin() and copyout() requesting quite large buffers from the fuzzers for some non-trivial functions like rump_init().

To-Do

• Improving the crash reproduction mechanism.
• Making the fuzzer smart by using grammar so that the arguments to syscalls are syntactically valid.
• Finding an optimal way to fetch data from fuzzer so that the reproducers are not sliced.

If the above improvements are done we get more coverage and if we are lucky enough lot more crashes.

Finally I'd like to thank my mentors Siddharth Muralee, Maciej Grochowski, Christos Zoulas for their guidance and Kamil Rytarowski for his constant support throughout the coding period.