DT_RPATH vs DT_RUNPATH

RPATH is used to embed a library search path in the executable. Since it takes precedence over default system library paths, it can be used both to specify the location of additional program libraries and to override system libraries.

Currently, RPATH can be embedded in executables using two tags: the “old” DT_RPATH tag and the “new” DT_RUNPATH tag. The existence of two tags comes from behavior exhibited by some operating systems (e.g. glibc systems): DT_RPATH used to take precedence over LD_LIBRARY_PATH, making it impossible to override the paths specified there. Therefore, a new DT_RUNPATH tag was added that comes after LD_LIBRARY_PATH in precedence. When both DT_RPATH and DT_RUNPATH are specified, the former is ignored.

On NetBSD, DT_RPATH does not take precedence over LD_LIBRARY_PATH. Therefore, there wasn't ever a need for DT_RUNPATH and the support for it (as alias to DT_RPATH) was added only very recently: on 2018-12-30.

Unlike GNU ld, LLD defaults to using “new” tag by default and therefore produces executables whose RPATHs do not work on older NetBSD versions. Given that using DT_RUNPATH on NetBSD has no real advantage, using the --disable-new-dtags option to suppress them is preferable.

More than two PT_LOAD segments

PT_LOAD segments are used to map executable image into the memory. Traditionally, GNU ld produces exactly two PT_LOAD segments: a RX text (code) segment, and a RW data segment. NetBSD dynamic loader (ld.elf_so) hardcodes the assumption of that design. However, lld sometimes produces an additional read-only data segment, causing the assertions to fail in the dynamic loader.

I have attempted to rewrite the memory mapping routine to allow for arbitrary number of segments. However, apparently my patch is just “a step in the wrong direction” and Joerg Sonnenberger is working on a proper fix.

Alternatively, LLD has a --no-rosegment option that can be used to suppress the additional segment and work around the problem.

Clang/LLD driver design issues

Both GCC and Clang use a design based on a front-end driver component. That is, the executable called directly by the user is a driver whose purpose is to perform initial command-line option and input processing, and run appropriate tools performing the actual work. Those tools may include the C preprocessor (cpp), C/C++ compiler (cc1), assembler, link editor (ld).

This follows the original UNIX principle of simple tools that perform a single task well, and a wrapper that combines those tools into complete workflows. Interesting enough, it makes it possible to keep all system-specific defaults and logic in a single place, without having to make every single tool aware of them. Instead, they are passed to those tools as command-line options.

This also accounts for simpler and more portable build system design. The gcc/clang driver provides a single high-level interface for performing a multitude of tasks, including compiling assembly files or linking executables. Therefore, the build system and the user do not need to be explicitly aware of low-level tooling and its usage. Not to mention it makes much easier to swap that tooling transparently.

For example, if you are linking an executable via the driver, it takes care of finding the appropriate link editor (and makes it easy to change it via -fuse-ld), preparing appropriate command-line options (e.g. if you do a -m32 multilib build, it sets the emulation for you) and passes necessary libraries to link (e.g. an appropriate standard C++ library when building a C++ program).

The clang toolchain considers LLD an explicit part of this workflow, and — unlike GNU ld — ld.lld is not really suitable for using stand-alone. For example, it does not include any standard search paths for libraries, expecting the driver to provide them in form of appropriate -L options. This way, all the logic responsible for figuring out the operating system used (including possible cross-compilation scenarios) and using appropriate paths is located in one component.

However, Joerg Sonnenberger disagrees with this and believes LLD should contain all the defaults necessary for it to be used stand-alone on NetBSD. Effectively, we have two conflicting designs: one where all logic is in clang driver, and the other where some of the logic is moved into LLD. At this moment, LLD is following the former assumption, and clang driver for NetBSD — the latter. As a result, neither using LLD directly nor via clang works out of the box on NetBSD; to use either, the user would have to pass all appropriate -L and -z options explicitly.

Fixing LLD to work with the current clang driver would require adding target awareness to LLD and changing a number of defaults for NetBSD based on the target used. However, LLD maintainer Rui Ueyama is opposed to introducing this extra logic specifically for NetBSD, and believes it should be added to the clang driver as for other platforms. On the other side, the NetBSD toolchain driver maintainer Joerg Sonnenberger blocks adding this to the driver. Therefore, we have reached a impasse that prevents LLD from working out of the box on NetBSD without local patches.

A work-in-progress implementation of the local target logic approach requested by Joerg can be seen in D56650. Afterwards, additional behavior can be enabled on NetBSD by using target triple properties such as in D56215 (which copies the libdir logic from clang).

For comparison, the same problem solved in a way consistent with other distributions (and rejected by Joerg) can be seen in D56932 (which updates D33726). However, in some cases it will require adding additional options to LLD (e.g. -z nognustack, D56554), and corresponding dummy switches in GNU ld.

Handling of indirect shared library dependencies

When starting a program, the dynamic loader needs to find and load all shared libraries listed via DT_NEEDED entries in order to obtain symbols needed by the program (functions, variables). Naturally, it also needs to process DT_NEEDED entries of those libraries to satisfy their symbol dependencies, and so on. As a result to this, the program can also reference symbols declared in dependencies of its DT_NEEDED libraries, that is its indirect dependencies.

While linking executables, link editors normally verify that all symbols can be resolved in one of the linked libraries. Historically, GNU ld followed the logic used by the dynamic loader and permitted symbols used by program to be present in either direct or indirect dependencies. However, GNU gold, LLD and newer versions of GNU ld use different logic and permit only symbols provided by the direct dependencies.

Let's take an example: you are writing a program that works with .zip files, and therefore you link -lzip. However, you also implement support for .gz files, and therefore call gzopen() provided by -lz which is also a dependency of libzip.so. Now, with old GNU ld versions you could just use -lzip since it would indirectly include libz.so. However, modern linkers will refuse to link it claiming that gzopen is undefined. You need to explicitly link -lzip -lz to resolve that.

Joerg Sonnenberger disagrees with this new behavior, and explicitly preserves the old behavior in GNU ld version used in NetBSD. However, LLD does not support the historical GNU ld behavior at all. It will probably become necessary to implement it from scratch to support NetBSD fully.

Summary

At this point, it seems that using LLD for the majority of regular packages is a goal that can be achieved soon. The main blocker right now is the disagreement between developers on how to proceed. When we can resolve that and agree on a single way forward, most of the patches become trivial.

Sadly, at this point I really do not see any way to convince either of the sides. The problem was reported in May 2017, and in the same month a fix consistent with all other platforms was provided. However, it is being blocked and LLD can not work out of the box.

Hopefully, we will able to finally find a way forward that does not involve keeping upstream clang driver for NetBSD incompatible with upstream LLD, and carrying a number of patches to LLD locally to make it work.

The further goals include attempting to build the kernel and complete userland using LLD, as well as further verifying compatibility of executables produced with various combinations of linker options (e.g. static PIE executables).