Working towards LLDB on i386

January 13, 2020 posted by Michał Górny

Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.

In February 2019, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support, extending NetBSD's ptrace interface to cover more register types and fix compat32 issues, fixing watchpoint and threading support.

Throughout December I've continued working on our build bot maintenance, in particular enabling compiler-rt tests. I've revived and finished my old patch for extended register state (XState) in core dumps. I've started working on bringing proper i386 support to LLDB.

Generic LLVM updates

Enabling and fixing more test suites

In my last report, I've indicated that I've started fixing test suite regressions and enabling additional test suites belonging to compiler-rt. So far I've been able to enable the following suites:

  • builtins library (alternative to libgcc)
  • profiling library
  • ASAN (address sanitizer, static and dynamic)
  • CFI (control flow integrity)
  • LSAN (leak sanitizer)
  • MSAN (memory sanitizer)
  • SafeStack (stack buffer overflow protection)
  • TSAN (thread sanitizer)
  • UBSAN (undefined behavior sanitizer)
  • UBSAN minimal
  • XRay (function call tracing)

In case someone's wondering how different memory-related sanitizers differ, here's a short answer: ASAN covers major errors that can be detected with approximate 2x slowdown (out-of-bounds accesses, use-after-free, double-free...), LSAN focuses on memory leaks and has almost no overhead, while MSAN detects unitialized reads with 3x slowdown.

The following test suites were skipped because of major known breakage, pending investigation:

  • generic interception tests (test runner can't find tests?)
  • fuzzer tests (many test failures, most likely as a side effect of LSAN integration)
  • clangd tests (many test failures)

The changes done to improve test suite status are:

Repeating the rationale for disabling ASLR/MPROTECT from my previous report: the sanitizers or tools in question do not work with the listed hardening features by design, and we explicitly make them fail. We are using paxctl to disable the relevant feature per-executable, and this makes it possible to run the relevant tests on systems where ASLR and MPROTECT are enabled globally.

This also included two builtin tests. In case of clear_cache_test.c, this is a problem with test itself and I have submitted a better MPROTECT support for clear_cache_test already. In case of enable_execute_stack_test.c, it's a problem with the API itself and I don't think it can be fixed without replacing it with something more suitable for NetBSD. However, it does not seem to be actually used by programs created by clang, so I do not think it's worth looking into at the moment.

Demise and return of LLD

In my last report, I've mentioned that we've switched to using LLD as the linker for the second stage builds. Sadly, this was only to discover that some of the new test failures were caused exactly by that.

As I've reported back in January 2019, NetBSD's dynamic loader does not support executables with more than two segments. The problem has not been fixed yet, and we were so far relying on explicitly disabling the additional read-only segment in LLD. However, upstream started splitting the RW segment on GNU RELRO, effectively restoring three segments (or up to four, without our previous hack).

This forced me to initially disable LLD and return to GNU ld. However, upstream has suggested using -znorelro recently, and we were enable to go back down to two segments and reenable it.

libc++ system feature list update

Kamil has noticed that our feature list for libc++ is outdated. We have missed indicating that NetBSD supports aligned_alloc(), timespec_get() and C11 features. I have updated the feature list.

Current build bot status

The stage 1 build currently fails as upstream has broken libc++ builds with GCC. Hopefully, this will be fixed after the weekend.

Before that, we had a bunch of failing tests: 7 related to profiling and 4 related to XRay. Plus, the flaky LLDB tests mentioned earlier.

Core dump XState support finally in

I was working on including full register set ('XState') in core dumps before my summer vacation. I've implemented the requested changes and finally pushed them. The patch set included four patches:

  1. Include XSTATE note in x86 core dump, including preliminary support for machine-dependent core dump notes.

  2. Fix alignment when reading core notes fixing a bug in my tests for core dumps that were added earlier,

  3. Combine x86 register tests into unified test function simplifying the test suite a lot (by almost a half of the original lines).

  4. Add tests for reading registers from x86 core dumps covering both old and new notes.

NetBSD/i386 support for LLDB

As the next step in my LLDB work, I've started working on providing i386 support. This covers both native i386 systems, and 32-bit executable support on amd64. In total, the combined amd64/i386 support covers four scenarios:

  1. 64-bit kernel, 64-bit debugger, 64-bit executable (native 64-bit).

  2. 64-bit kernel, 64-bit debugger, 32-bit executable.

  3. 64-bit kernel, 32-bit debugger, 32-bit executable.

  4. 32-bit kernel, 32-bit debugger, 32-bit executable (native 32-bit).

Those cases are really different only from kernel's point-of-view. For scenarios 1. and 2. the debugger is using 64-bit ptrace API, while in cases 3. and 4. it is using 32-bit ptrace API. In case 2., the application runs via compat32 and the kernel fits its data into 64-bit ptrace API. In case 3., the debugger runs via compat32.

Technically, cases 1. and 2. are already covered by the amd64 code in LLDB. However, for user's convenience LLDB needs to be extended to recognize 32-bit processes on NetBSD and adjust the data obtained from ptrace to 32-bit executables. Cases 3. and 4. need to be covered via making the code build on i386.

Other LLDB plugins implement this via creating separate i386 and amd64 modules, then including 32-bit branch in amd64 that reuses parts of i386 code. I am following suit with that. My plan is to implement 32-bit process support for case 2. first, then port everything to i386.

So far I have implemented the code to recognize 32-bit processes and I have started implementing i386 register interface that is meant to map data from 64-bit ptrace register dumps. However, it does not seem to map registers correctly at the moment and I am still debugging the problem.

Future plans

As mentioned above, I am currently working on providing support for debugging 32-bit executables on amd64. Afterwards, I am going to work on porting LLDB to run on i386.

I am also tentatively addressing compiler-rt test suite problems in order to reduce the number of build bot failures. I also need to look into remaining kernel problems regarding simultaneous delivery of signals and breakpoints or watchpoints.

Furthermore, I am planning to continue with the items from the original LLDB plan. Those are:

  1. Add support to backtrace through signal trampoline and extend the support to libexecinfo, unwind implementations (LLVM, nongnu). Examine adding CFI support to interfaces that need it to provide more stable backtraces (both kernel and userland).

  2. Add support for aarch64 target.

  3. Stabilize LLDB and address breaking tests from the test suite.

  4. Merge LLDB with the base system (under LLVM-style distribution).

This work is sponsored by The NetBSD Foundation

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:



Post a Comment:
  • HTML Syntax: NOT allowed