Adapting TriforceAFL for NetBSD, Part 2

Prepared by Akul Pillai as part of GSoC 2019.

I have been working on adapting TriforceAFL for NetBSD kernel syscall fuzzing. This blog post summarizes the work done until the second evaluation.

For work done during the first coding period, check out this post.

Input Generation

For a feedback driven mutation based fuzzer that is TriforceAFL, fuzzing can be greatly improved by providing it with proper input test cases. The fuzzer can then alter parts of the valid input leading to more coverage and hopefully more bugs. The TriforceNetBSDSyscallFuzzer itself was a working fuzzer at the end of the first evaluation, but it was missing some proper input generation for most of the syscalls.
A greater part of the time during this coding period was spent adding and testing basic templates for a majority of NetBSD syscalls, scripts have also been added for cases where more complex input generation was required. This should now allow the fuzzer to find bugs it previously could not have.

Templates for 160 of the 483 syscalls in NetBSD have been added, below is the complete list:

1 exit, 2 fork, 3 read, 4 write, 5 open, 6 close, 7 compat_50_wait4, 8 compat_43_ocreat, 9 link, 10 unlink, 12 chdir, 13 fchdir, 14 compat_50_mknod, 15 chmod, 16 chown, 17 break, 19 compat_43_olseek, 20 getpid, 22 unmount, 23 setuid, 24 getuid, 25 geteuid, 26 ptrace, 33 access, 34 chflags, 35 fchflags, 36 sync, 37 kill, 39 getppid, 41 dup, 42 pipe, 43 getegid, 44 profil, 45 ktrace, 47 getgid, 49 __getlogin, 50 __setlogin, 51 acct, 55 compat_12_oreboot, 56 revoke, 57 symlink, 58 readlink, 59 execve, 60 umask, 61 chroot, 62 compat_43_fstat43, 63 compat_43_ogetkerninfo, 64 compat_43_ogetpagesize, 66 vfork, 73 munmap, 78 mincore, 79 getgroups, 80 setgroups, 81 getpgrp, 82 setpgid, 83 compat_50_setitimer, 86 compat_50_getitimer, 89 compat_43_ogetdtablesize, 90 dup2, 95 fsync, 96 setpriority, 97 compat_30_socket, 100 getpriority, 106 listen, 116 compat_50_gettimeofday, 117 compat_50_getrusage, 120 readv, 121 writev, 122 compat_50_settimeofday, 123 fchown, 124 fchmod, 126 setreuid, 127 setregid, 128 rename, 131 flock, 132 mkfifo, 134 shutdown, 135 socketpair, 136 mkdir, 137 rmdir, 140 compat_50_adjtime, 147 setsid, 161 compat_30_getfh, 165 sysarch, 181 setgid, 182 setegid, 183 seteuid, 191 pathconf, 192 fpathconf, 194 getrlimit, 195 setrlimit, 199 lseek, 200 truncate, 201 ftruncate, 206 compat_50_futimes, 207 getpgid, 209 poll, 231 shmget, 232 compat_50_clock_gettime, 233 compat_50_clock_settime, 234 compat_50_clock_getres, 240 compat_50_nanosleep, 241 fdatasync, 242 mlockall, 243 munlockall, 247 _ksem_init, 250 _ksem_close, 270 __posix_rename, 272 compat_30_getdents, 274 lchmod, 275 lchown, 276 compat_50_lutimes, 289 preadv, 290 pwritev, 286 getsid, 296 __getcwd, 306 utrace, 344 kqueue, 157 compat_20_statfs, 158 compat_20_fstatfs, 416 __posix_fadvise50, 173 pread, 174 pwrite, 197 mmap, 462 faccessat, 463 fchmodat, 464 fchownat, 461 mkdirat, 459 mkfifoat, 460 mknodat, 468 openat, 469 readlinkat, 458 renameat, 470 symlinkat, 471 unlinkat, 453 pipe2, 467 utimensat

A separate script (targ/gen2.py) generating tricker input cases was added for the following:

104 bind, 105 setsockopt, 118 getsockopt, 98 connect, 30 accept, 31 getpeername, 32 getsockname, 133 sendto, 29 recvfrom, 21 compat_40_mount, 298 compat_30_fhopen 299 compat_30_fhstat, 300 compat_20_fhstatfs, 93 compat_50_select, 373 compat_50_pselect, 345 compat_50_kevent, 92 fcntl, 74 mprotect, 203 mlock, 273 minherit, 221 semget, 222 semop, 202 __sysctl

Reproducibility

The fuzzer uses the simplest way to reproduce a crash which is by storing the exact input for the test case that resulted in a crash. This input can then be passed to the driver program, which will be able to parse the input and execute the syscalls that were executed in the order that they were executed.
A better prototype reproducer generator has now been added to the fuzzer that provides us with human readable and executable C code. This C code can be compiled and executed to reproduce the crash.
To generate the reproducers simply run the genRepro script from the targ directory. If everything goes right, the C reproducers will now be available in targ/reproducers, in separate files as follows:

// id:000009,sig:00,src:005858+002155,op:splice,rep:8
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mincore, 0x/* removed from the report due to security concerns */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000010,sig:00,src:005859+004032,op:splice,rep:2
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000011,sig:00,src:005859+004032,op:splice,rep:4
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

The reproducers currently do not include the allocated memory and such, so not all reproducers will work. More improvements are to come, but this will hopefully make analysis of the crashes easier.

Fuzzing

The fuzzer was run for ~4 days. We were getting ~50 execs/sec on a single core. Please note that this is using qemu with softemu (TCG) as hardware acceleration cannot be used here. During this period the fuzzer detected 23 crashes that it marked as unique. Not all of these were truly unique, there is the scope of adding a secondary filter here to detect truly unique crashes. Most of them were duplicates of the crashes highlighted below:

compat_43_osendmsg - tcp_output: no template

call 114 - compat_43_osendmsg
arg 0: argStdFile 4 - type 12
arg 1: argVec64 77d0549cc000 - size 4
arg 2: argNum 8003

read 83 bytes, parse result 0 nrecs 1
syscall 114 (4, 77d0549cc000, 8003)

[ 191.8169124] panic: tcp_output: no template
[ 191.8169124] cpu0: Begin traceback...
[ 191.8269174] vpanic() at netbsd:vpanic+0x160
[ 191.8269174] snprintf() at netbsd:snprintf
[ 191.8269174] tcp_output() at netbsd:tcp_output+0x2869
[ 191.8385864] tcp_sendoob_wrapper() at netbsd:tcp_sendoob_wrapper+0xfe
[ 191.8469824] sosend() at netbsd:sosend+0x6e3
[ 191.8469824] do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x231
[ 191.8469824] do_sys_sendmsg() at netbsd:do_sys_sendmsg+0xac
[ 191.8569944] compat_43_sys_sendmsg() at netbsd:compat_43_sys_sendmsg+0xea
[ 191.8569944] sys___syscall() at netbsd:sys___syscall+0x74
[ 191.8655484] syscall() at netbsd:syscall+0x181
[ 191.8655484] --- syscall (number 198) ---
[ 191.8655484] 40261a:
[ 191.8655484] cpu0: End traceback...
[ 191.8655484] fatal breakpoint trap in supervisor mode
[ 191.8655484] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x7f7795402000 ilevel 0x4 rsp 0xffffc58032d589f0
[ 191.8655484] curlwp 0xfffff7e8b1c63220 pid 700.1 lowest kstack 0xffffc58032d55
2c0
Stopped in pid 700.1 (driver) at        netbsd:breakpoint+0x5:  leave

mincore - uvm_fault_unwire_locked: address not in map

call 78 - mincore
arg 0: argNum d000000
arg 1: argNum 7600000000000000
arg 2: argNum 1b0000000000
read 65 bytes, parse result 0 nrecs 1
syscall 78 (d000000, 7600000000000000, 1b0000000000)

[ 141.0578675] panic: uvm_fault_unwire_locked: address not in map
[ 141.0578675] cpu0: Begin traceback...
[ 141.0691345] vpanic() at netbsd:vpanic+0x160
[ 141.0691345] snprintf() at netbsd:snprintf
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire+0x29
[ 141.0774205] sys_mincore() at netbsd:sys_mincore+0x23c
[ 141.0884435] sys___syscall() at netbsd:sys___syscall+0x74
[ 141.0884435] syscall() at netbsd:syscall+0x181
[ 141.0884435] --- syscall (number 198) ---
[ 141.0996065] 40261a:
[ 141.0996065] cpu0: End traceback...
[ 141.0996065] fatal breakpoint trap in supervisor mode
[ 141.0996065] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x761548094000 ilevel 0 rsp 0xffff870032e48d90
[ 141.0996065] curlwp 0xffff829094e51b00 pid 646.1 lowest kstack 0xffff870032e45
2c0
Stopped in pid 646.1 (driver) at        netbsd:breakpoint+0x5:  leave

extattrctl - KASSERT fail

call 360 - extattrctl
arg 0: argBuf 7b762b514044 from 2 bytes
arg 1: argNum ff8001
arg 2: argFilename 7b762b515020 - 2 bytes from /tmp/file0
arg 3: argNum 0
arg 4: argNum 0
arg 5: argNum 2100000000
read 59 bytes, parse result 0 nrecs 1
syscall 360 (7b762b514044, ff8001, 7b762b515020, 0, 0, 2100000000)

[ 386.4528838] panic: kernel diagnostic assertion "fli->fli_trans_cnt == 0" fail
ed: file "src/sys/kern/vfs_trans.c", line 201
[ 386.4528838] cpu0: Begin traceback...
[ 386.4528838] vpanic() at netbsd:vpanic+0x160
[ 386.4648968] stge_eeprom_wait.isra.4() at netbsd:stge_eeprom_wait.isra.4
[ 386.4724138] fstrans_lwp_dtor() at netbsd:fstrans_lwp_dtor+0xbd
[ 386.4724138] exit1() at netbsd:exit1+0x1fa
[ 386.4724138] sys_exit() at netbsd:sys_exit+0x3d
[ 386.4832968] syscall() at netbsd:syscall+0x181
[ 386.4832968] --- syscall (number 1) ---
[ 386.4832968] 421b6a:
[ 386.4832968] cpu0: End traceback...
[ 386.4832968] fatal breakpoint trap in supervisor mode
[ 386.4944688] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0xffffc100324bd000 ilevel 0 rsp 0xffffc10032ce9dc0
[ 386.4944688] curlwp 0xfffff6278e2fc240 pid 105.1 lowest kstack 0xffffc10032ce6
2c0
Stopped in pid 105.1 (driver) at        netbsd:breakpoint+0x5:  leave

pkgsrc Package

Lastly, the TriforceNetBSDSyscallFuzzer has now been made available in the form of a pkgsrc package in pkgsrc/wip as triforcenetbsdsyscallfuzzer. The package will require wip/triforceafl which was ported earlier.
All other changes mentioned can be found in the github repo.

script(1) recording

A typescript recording of a functional TriforceAFL fuzzer setup and execution is available here. Download it and replay it with script -p.

Future Work

Work that remains to be done include:

• Restructuring of files
  The file structure needs to be modified to suit the specific case of the Host and Target being the same OS. Right now, files are separated into Host and Target directories, this is not required.
• Testing with Sanitizers enabled
  Until now the fuzzing done was without using KASAN or kUBSAN. Testing with them enabled and fuzzing with them will be of major focus in the third coding period.
• Improving the 'reproducer generator'
  There is some scope of improvement for the prototype that was added. Incremental updates to it are to be expected.
• Analysis of crash reports and fixing bugs
• Documentation

Summary

So far, the TriforceNetBSDSyscallFuzzer has been made available in the form of a pkgsrc package with the ability to fuzz most of NetBSD syscalls. In the final coding period of GSoC. I plan to analyse the crashes that were found until now. Integrate sanitizers, try and find more bugs and finally wrap up neatly with detailed documentation.

Last but not least, I would like to thank my mentor, Kamil Rytarowski for helping me through the process and guiding me. It has been a wonderful learning experience so far!

[2 comments]