LLVM, Clang and compiler-rt support enhancements

In the last month I started with upstream of the code for sanitizers: the common layer and ubsan. I worked also on the elimination of unexpected failures in LLVM and Clang. I've managed to achieve, with a pile of local patches, the number of 0 unexpected bugs within LLVM (check-llvm) and 3 unexpected bugs within Clang (check-clang) (however these ones were caused by hardcoded environment -lstdc++ vs -lc++). The number of failures in sanitizers (check-sanitizer) is also low, it's close to zero.

LLVM

In order to achieve the goals of testability concerning the LLVM projects, I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree. The set of these projects is composed of: llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra. These were required to build and execute test-suites in the LLVM's projects. Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang, however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package in order to develop the work environment with the ability to build and execute unit tests.

There were four threads within LLVM:

• Broken std::call_once with libstdc++. This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::call_once. I've discovered that the llvm::call_once workaround isn't sufficient for the whole LLVM functionality, as std::call_once can be called internally inside the libstdc++ libraries - like within the C++11 futures interface. This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker.
• Unportable shell construct hardcoded in tests ">&". This has been fixed upstream.
• LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X. This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces. This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements. I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream. My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X.
• Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt.

The LLVM JIT API is the last cause of unexpected failures in check-llvm. This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests.

Clang

I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64. This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt.

compiler-rt

The current compiler-rt tasks can be divided into:

1. upstream sanitizer common code shared with POSIX platforms
2. upstream sanitizer common code shared with Linux and FreeBSD
3. upstream sanitizer common code shared with FreeBSD
4. upstream sanitizer common code specific to NetBSD
5. build, execute and pass tests for sanitizer common code in check-santizer

This means that ubsan, asan and the rest of the specific sanitizers wait in queue.

All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one.

The last point with check-sanitizer unveiled so far two generic bugs on NetBSD:

• Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls.
• Honor PTHREAD_DESTRUCTOR_ITERATIONS in libpthread.

These bugs are not strictly real bugs, but they were introducing needless differences with other modern POSIX systems. The fixes were introduced by Christos Zoulas and backported to NetBSD-8.

Plan for the next milestone

I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code. The roadmap for the next month is to continue working on the goals of the previous months. std::call_once is an example that every delayed bug keeps biting again and again in future.

LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT. There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0).

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate [0 comments]