Followup from last month and buildbot updates

By the end of February, I was working on fixing the most urgent test failures in order to resume continuous testing on buildbot. In early March, I was able to get all the necessary fixes committed. The list includes:

• fixing tests to always use libc++ to avoid libstdc++-related breakage: r355273,

• enabling passing -pthread flag: r355274,

• enabling support for finding libc++ headers relatively to the executable location: r355282,

• fixing additional case of bind/connect address mismatch: r355285,

• passing appropriate -L and -Wl,-rpath flags for tests: r355502 with followup test fix in r355510.

The commit series also included an update for the code finding main executable location to use sysctl(): r355283. However, I've reverted it afterwards as it did not work reliably: r355302. Since this was neither really necessary nor unanimously considered correct, I've abandoned the idea.

Once those initial issues were fixed, I was able to enable the full LLDB test suite on the buildbot. Based on the initial results, I updated the list of tests known to fail on NetBSD: r355320, and later r355774, and start looking into ‘flaky’ tests.

Tests are called flaky if they can either pass or fail unpredictably — usually as a result of race conditions, timeouts and other events that may depend on execution order, system load, etc. The LLDB test suite provides a workaround for flaky tests — through executing them multiple times, and requiring only one of the runs to pass.

I have initially attempted to take advantage of this, committing r355830, then r355838. However, this approach turned out to be suboptimal. Sadly, marking more tests flaky only yielded more failures than previously — presumably because of the load increased through rerunning failing tests. Therefore, I've decided it more prudent to instead focus on finding the root issue.

Currently we are facing a temporary interruption in our buildbot service. We will restore it as soon as possible.

Threaded and AArch64 core file support

The next task on the list was to finish work on improving NetBSD core file support that was started by Kamil Rytarowski from almost two years ago. This involved rebasing his old patches after major upstream refactoring, adding tests and addressing remarks from upstream.

Firstly, I've addressed support for core files created from threaded programs. The new code itself landed as r355736. However, one of the tests was initially broken as it relied on symbol names provided by libc, and therefore failed on non-NetBSD systems. After initially disabling the test, I've finally fixed it by refactoring the code to fail in regular program function rather than libc call: r355786.

Secondly, I've added support for core files from AArch64 systems. To achieve this, I have set up a QEMU VM with a lot of help from Jared McNeill. For completeness, I include his very helpful instructions here:

1. Fetch http://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/latest/QEMU-AARCH64/RELEASE_GCC49/QEMU_EFI.fd

2. Fetch and uncompress latest arm64.img.gz from http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/binary/gzimg/

3. Use qemu-img resize arm64.img <newsize> to expand the image for your needs

4. Start qemu:

   SMP=4
   MEM=2g
   qemu-system-aarch64 -M virt -machine gic-version=3 -cpu cortex-a53 -smp $SMP -m $MEM \
      -drive if=none,file=arm64.img,id=hd0 -device virtio-blk-device,drive=hd0 \
      -netdev type=user,id=net0 -device virtio-net-device,netdev=net0,mac=00:11:22:33:44:55 \
      -bios QEMU_EFI.fd \
      -nographic

With this approach, I've finally been able to run NetBSD/arm64 via QEMU. I used it to create matching core dumps for the tests, and afterwards implemented AArch64 support: r357399.

Other improvements

During the month, upstream has introduced a new SBReproducer module that wrapped most of the LLDB API. The original implementation involved generating the wrappers for all modules in a single file. Combined with heavy use of templates, building this file caused huge memory consumption, making it impossible to build on systems with 4 GiB of RAM. I've discussed possible solutions with upstream and finally implemented one splitting and moving wrappers to individual modules: r356481.

Furthermore, as an effort to reduce flakiness of tests, I've worked on catching and fixing more functions potentially interrupted via signals (EINTR): r356703, with fixup in r356960. Once the LLVM bot will be back, we will check whether that solved our flakiness problem.

LLVM 8 release

Additionally, during the month LLVM 8.0.0 was finally released. Following Kamil's request, by the end of the month I've started working on updating NetBSD src for this release. This is still a work in progress, and I've only managed to get LLVM and Clang to build (with other system components failing afterwards). Nevertheless, if you're interested the current set of changes can be seen on my GitHub fork of netbsd/src, llvm8 branch. The URL for comparison is: https://github.com/NetBSD/src/compare/6e11444..mgorny:llvm8

Please note that for practical reasons this omits the commit updating distributed LLVM and Clang sources.

Future plans

The plans for the nearest future include finishing the efforts mentioned here and working with others on resuming the buildbot. I have also discussed with Pavel Labath and he suggested working on improving error reporting to help us resolve current and future test failures.

The next milestones in the LLDB development plan are:

7. Add support for FPU registers support for NetBSD/i386 and NetBSD/amd64.

8. Support XSAVE, XSAVEOPT, ... registers in core(5) files on NetBSD/amd64.

This work is sponsored by The NetBSD Foundation

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

http://netbsd.org/donations/#how-to-donate