Enhancing Syzkaller support for NetBSD, Part 1

Prepared by Siddharth Muralee(@R3x) as a part of Google Summer of Code 2019

As a part of Google Summer of Code 19, I am working on improving the support for Syzkaller kernel fuzzer. Syzkaller is an unsupervised coverage-guided kernel fuzzer, that supports a variety of operating systems including NetBSD. This report details the work done during the first coding period.

Syzkaller

Initially, Syzkaller was developed with Linux kernel fuzzing in mind, but now it's being extended to support other OS kernels as well. The main target of the Syzkaller fuzzer is the system call layer.

Thanks to Google, we now have a 24/7 continuous fuzzing instance on a Google Cloud engine for NetBSD managed by Syzbot, a sub-system of the Syzkaller fuzzer. Syzbot maintains a dashboard where it reports all the bugs that were found. Syzbot also maintains a mailing list to report bugs found.

Syzbot is currently fuzzing NetBSD-HEAD which it updates from the Github mirror.

You can go through the Syzkaller documentation or take a look at my slides from Troopers 19 for learning more about Syzkaller.

Automated Image creation

Syzkaller required NetBSD images with ssh-keys for continous fuzzing. Due to frequent updates to HEAD sometimes images become unusable. So I decided to automate the process of creating NetBSD images with ssh-keys and appropriate settings.

Initial attempts with packer by Hashicorp failed because the boot sequence was controlled by waits and not output matching. So I wrote a shell script which basically adds a wrapper around anita to do the job.

Kcov(4) support

Kernel Code Coverage(KCov) is a compiler instrumented feature which helps us in finding the code paths executed inside the kernel for a certain set of system calls. An initial port was done by me with modifications by @kamil and @maxv.

Syzkaller uses coverage for modifying and mutating the arguments of syscalls. Coverage information for NetBSD is publicly available.

Sanitizers Support

Sanitizers are compiler instrumented tools to improve code correctness. Currently, NetBSD supports Kernel Address Sanitizer(KASAN) and Kernel Undefined behaviour Sanitizer(KUBSAN).

We use the Sanitizers to increase the chances of finding bugs. Syzkaller now compiles kernels with the Sanitizers.

Report Generation and Symbolization

Syzkaller logs the console and in the event of a crash it records the log to find out details about the crash, these details are then used to classify the crash and create a report.

For better crash reports, we decided to enable a ddb(4) shell on event of a kernel panic. This allowed us to print backtraces, details of locks and processes.

Also I added support for better Symbolization of the reports. Symbolization is adding more details in the crash report to make them easier for developers to go through. Currently we have added file names and line numbers for functions in the crash based on the kernel object (netbsd.gdb).

Initial backtrace :

do_ptrace() at netbsd:do_ptrace+0x33d
sys_ptrace() at netbsd:sys_ptrace+0x71
sys_syscall() at netbsd:sys_syscall+0xf5

After Symbolization :

do_ptrace() at netbsd:do_ptrace+0x33d sys/kern/sys_ptrace_common.c:1430
sys_ptrace() at netbsd:sys_ptrace+0x71 sys/kern/sys_ptrace.c:218
sys_syscall() at netbsd:sys_syscall+0xf5 sy_call sys/sys/syscallvar.h:65 [inline]

Syscall Coverage Improvements

Syzkaller uses a pseudo-formal grammar for system calls. It uses the same to build programs to fuzz the kernel. The files related to the same are stored at sys/netbsd in the syzkaller repo. These files were a rough copy from the linux files with whatever was unable to compile removed.

We had to review all the existing syscall descriptions, find the missing ones and add them.

I wrote a python script which would help in finding out existing syscall descriptions and match them with the NetBSD description. The script also finds missing syscalls and logs them.

I have listed the system calls that are currently fuzzed with Syzkaller.

We are currently working on adding and improving descriptions for the more important syscalls. If you would like to see a NetBSD system call fuzzed you can reach out to us.

Summary

During the last month, I was focusing on improving support for NetBSD. I have managed to complete the tasks that we had planned for the first evaluation.

For the next coding period (28th June - 22nd July) I will be working on adding support for fuzzing the Network layer.

Last but not least, I want to thank my mentors, @kamil and @cryo for their useful suggestions and guidance. I would also like to thank Dmitry Vyukov, Google for helping with any issues faced with regard to Syzkaller. Finally, thanks to Google to give me a good chance to work with NetBSD community.