Bookmarks

Feeds

Summary of the preliminary LLDB support project

January 23, 2017 posted by Kamil Rytarowski

Operating systems can be called monitors as they handle system calls from userland processes. A similar task is performed by debuggers as they implement monitors for traced applications and interpret various events that occurred in tracees and are messaged usually with signals to their tracers. During this month I have started a new Process Plugin within LLDB to incept NativeProcessNetBSD - copied from NativeProcessLinux - implementing basic functionality and handling all the needed events in the MonitorCallback() function. To achieve these tasks, I had to add a bunch of new ptrace(2) interfaces in the kernel to cover all that is required by LLDB monitors. The current Process Plugin for NetBSD is capable to start a process, catch all the needed events correctly and if applicable resume or step the process.

What has been done in NetBSD

1. Verified the full matrix of combinations of wait(2) and ptrace(2) in the following test-cases

verify basic PT_GET_SIGINFO call for SIGTRAP from tracee
verify basic PT_GET_SIGINFO and PT_SET_SIGINFO calls without modification of SIGINT from tracee
verify basic PT_GET_SIGINFO and PT_SET_SIGINFO calls with setting signal to new value
detect SIGTRAP TRAP_EXEC from tracee
verify single PT_STEP call with signal information check
verify that fork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_FORK and reports correct signal information
verify that vfork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_VFORK_DONE
verify that vfork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_FORK | PTRACE_VFORK_DONE
verify that PTRACE_VFORK in EVENT_MASK is preserved
verify that PTRACE_VFORK_DONE in EVENT_MASK is preserved
verify that PTRACE_LWP_CREATE in EVENT_MASK is preserved
verify that PTRACE_LWP_EXIT in EVENT_MASK is preserved
verify that thread creation is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_LWP_CREATE
verify that thread termination is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_LWP_EXIT

2. GNU libstdc++ std::call_once bug investigation

The LLVM toolchain internally uses several std::call_once calls, within LLVM and LLDB codebase. The GNU libstdc++ library implements C++11 std::call_once with wrapping pthread_once(3) and a function pointer passed as a TLS (Thread Local Storage) variable. Currently this setup isn't functional with shared linking on NetBSD and apparently on few other systems and hardware platforms. This issue is still unresolved and its status is traced in PR 51139. As of now, there is a walkaround originally prepared for NetBSD, and adapted for others - namely llvm::call_once.

3. Improving documentation and other minor system parts

rename the SIGPOLL signal to SIGIO in the documentation of siginfo(2)
replace SIGPOLL references with SIGIO in comments of <sys/siginfo.h>
document SI_NOINFO in siginfo(2)
document SI_LWP in siginfo(2)
document SI_QUEUE in siginfo(2)
document SI_MESGQ in siginfo(2)
document TRAP_EXEC in siginfo(2)
document TRAP_CHLD in siginfo(2)
document TRAP_LWP in siginfo(2)
reference siginfo(2) for a SIGCHLD signal in ptrace(2)
remove unused macro for ATF_TP_ADD_TC_HAVE_DBREGS in src/tests/kernel/t_ptrace_wait.h
cleanup commented out code after revert of racy vfork(2) commit from year 2012
remove stub code to redefine PTRACE_FORK in ATF tests
... and several microimprovements in the codebase

4. Documentation of ptrace(2) and explanation how debuggers work

document addr and data argument usage for PT_GET_PROCESS_STATE
document PT_SET_SIGINFO and PT_GET_SIGINFO
explain execve(2) handling and behavior, SIGTRAP & TRAP_EXEC
reference PaX MPROTECT restrictions for debuggers
explain software breakpoints handling and behavior, SIGTRAP & TRAP_BKPT
explain single step behavior, SIGTRAP & TRAP_TRACE
explain that PT_TRACE_ME does not send a SIGSTOP signal
list predefined MI symbols for help debuggers in port specific headers
document the current behavior of TRAP_CHLD
add more notes on PTRACE_FORK events
document PTRACE_VFORK and PTRACE_VFORK_DONE
document PTRACE_LWP_CREATE and PTRACE_LWP_EXIT
import HISTORY notes from FreeBSD

5. Introduction of new siginfo(2) codes for SIGTRAP

TRAP_EXEC - catch exec() events transforming the calling process into a new process
TRAP_CHLD - process child trap (fork, vfork and vfork done events)
TRAP_LWP - process thread traps (birth, termination)
TRAP_HWWPT - process hardware assisted watchpoints

6. New ptrace(2) interfaces

There were added new interfaces to the native ptrace(2) NetBSD API.

Interface to introspect and fake signal information

The PT_GET_SIGINFO interface is designed to read signal information sent to tracee. A program can also fake a signal to be passed to a debugged process, with included destination thread or the whole process. A debugger requires detailed signal information in order to distinguish the exact trap reason easily and precisely, for example whether a program stopped on a software defined breakpoint or a single step trap.

This interface is modeled after the Linux specific calls: PTRACE_GETSIGINFO and PTRACE_SETSIGINFO, but adapted for the NetBSD use-case. FreeBSD currently has no way to fake signal information.

I have modeled this interface to be most efficient in terms of determination what exact thread received the signal. Thanks to it, a process does not need to examine each thread separately and performs only a single ptrace(2) call. This makes a significant difference in a massively threaded software.

The signal information accessors introduce a new structure ptrace_siginfo:

/*
 * Signal Information structure
 */
typedef struct ptrace_siginfo {
       siginfo_t       psi_siginfo;    /* signal information structure */
       lwpid_t         psi_lwpid;      /* destination LWP of the signal
                                        * value 0 means the whole process
                                        * (route signal to all LWPs) */
} ptrace_siginfo_t;

I've included in the <sys/ptrace.h> header required file to define the siginfo_t type - <sys/siginfo.h>. This makes sure that no existing software will break during build due to a missing type definition.

Interface to monitor vfork(2) operations

Forking a process is one of the fundamental design features of a UNIX operating system. There are two basic types of forks: fork(2) and vfork(2) ones. The fork(2) one is designed to spawn a mostly independent child from parent's memory layout, however this operation is costy. In order to address it and optimize the forking operation there is vfork(2) which shares address space with its parent. From the ptrace(2) point of view, the difference between fork(2) and vfork(2) calls is whether a parent giving birth to its child is suspended waiting on child termination or exec() operation or not.

Currently there is an interface to monitor fork(2) operations - PTRACE_FORK - however NetBSD missed the vfork(2) ones. I've added two new functions: PTRACE_VFORK and PTRACE_VFORK_DONE. The former is supposed to notify why a parent gives a vfork(2)-like birth to its child and later when a child is born with vfork(2) from its parent - perfoming a handshake with two SIGTRAP signals emitted from the kernel. Once the parent is resuming it gets a notification for vfork(2) done.

These events throw SIGTRAP signal with the TRAP_CHLD property. Currently PTRACE_VFORK is a stub and it's planned to be fully implemented later.

Interface to monitor thread operations

A debugger requires an interface to monitor thread birth and termination. This is needed in order to properly track the thread list of a process and ensure that every thread has for example applied watchpoints.

I've added two events:

PTRACE_LWP_CREATE - to report thread births,
PTRACE_LWP_EXIT - to report thread termination.

This interface reuses the EVENT_MASK and PROCESS_STATE interface. It means that it shares these calls with PTRACE_FORK, PTRACE_VFORK and PTRACE_VFORK_DONE.

To achieve this goal, I've changed the following structure:

typedef struct ptrace_state {
        int     pe_report_event;
        pid_t   pe_other_pid;
} ptrace_state_t;

typedef struct ptrace_state {
        int     pe_report_event;
        union {
                pid_t   _pe_other_pid;
                lwpid_t _pe_lwp;
        } _option;
} ptrace_state_t;

#define pe_other_pid    _option._pe_other_pid
#define pe_lwp          _option._pe_lwp

This change keeps the size of ptrace_state_t unchanged as both pid_t and lwpid_t are defined as an int32_t-like integer. New struct form should not break existing software and be source and binary compatible with it.

I've introduced a new SIGTAP type for thread events: TRAP_LWP.

Hardware assisted watchpoints

I've introduced a few changes to the current interface. One of them is allowing to mix single-step operation with enabled hardware assisted watchpoints. The other one was added new extension pw_type to the ptrace_watchpoint structure.

6. Updated doc/TODO.ptrace entries

The current state of TODO.ptrace - after several updates - is as follows:

verify ppid of core dump generated with PT_DUMPCORE it must point to the real parent, not tracer
adapt OpenBSD regress test (regress/sys/ptrace/ptrace.c) for the ATF context
add new ptrace(2) calls to lock (suspend) and unlock LWP within a process
add PT_DUMPCORE tests in the ATF framework
add ATF tests for PT_WRITE_I and PIOD_WRITE_I - test mprotect restrictions
add ATF tests for PIOD_READ_AUXV
once the API for hardware watchpoints will stabilize, document it
add tests for the procfs interface covering all functions available on the same level as ptrace(2)
add support for PT_STEP, PT_GETREGS, PT_SETREGS, PT_GETFPREGS, PT_SETFPREGS in all ports
integrate all ptrace(2) features in gdb
add ptrace(2) NetBSD support in LLDB
add proper implementation of PTRACE_VFORK for vfork(2)-like events
remove exect(3) - there is no usecase for it
refactor pthread_dbg(3) to only query private pthread_t data, otherwise it duplicates ptrace(2) interface and cannot cover all types of threads
add ATF tests for SIGCHLD
add ATF tests for PT_SYSCALL and PT_SYSCALLEMU

Features in ELF, DWARF, CTF, DTrace are out of scope for the above list.

7. Future directions

After research and testing the current watchpoint interface, I've realized that it's impossible (impractically complicated) to pretend to have a "safe" watchpoint interface inside the kernel, as the current one isn't safe from undefined behavior even on stock amd64. I've decided to revert this code and introduce PT_GETDBREGS and PT_SETDBREGS restricted to INSECURE secure level mode. The good side of this change is that there is already part of the code needed in the kernel, I have a local draft introducing this interface and it will be easier to integrate with LLDB, as Linux and FreeBSD keep having the same interface.

What has been done in LLDB

I work on the LLDB port inside the pkgsrc-wip repository, in the lldb-netbsd package.

To summarize the changes, there were so far 84 commits in this directory. The overall result is a list of 26 patched or added files. The overall diff's length is 3539 lines.

$ wc -l patches/patch-*                                                               
      12 patches/patch-cmake_LLDBDependencies.cmake
      17 patches/patch-cmake_modules_AddLLDB.cmake
      14 patches/patch-include_lldb_Host_netbsd_HostThreadNetBSD.h
      30 patches/patch-include_lldb_Host_netbsd_ProcessLauncherNetBSD.h
      12 patches/patch-source_CMakeLists.txt
      12 patches/patch-source_Host_CMakeLists.txt
      60 patches/patch-source_Host_common_Host.cpp
      13 patches/patch-source_Host_common_NativeProcessProtocol.cpp
      21 patches/patch-source_Host_netbsd_HostThreadNetBSD.cpp
     175 patches/patch-source_Host_netbsd_ProcessLauncherNetBSD.cpp
      23 patches/patch-source_Host_netbsd_ThisThread.cpp
      24 patches/patch-source_Initialization_SystemInitializerCommon.cpp
     803 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.cpp
     141 patches/patch-source_Plugins_Platform_NetBSD_PlatformNetBSD.h
      12 patches/patch-source_Plugins_Process_CMakeLists.txt
      13 patches/patch-source_Plugins_Process_NetBSD_CMakeLists.txt
    1392 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.cpp
     188 patches/patch-source_Plugins_Process_NetBSD_NativeProcessNetBSD.h
     393 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.cpp
      92 patches/patch-source_Plugins_Process_NetBSD_NativeThreadNetBSD.h
      13 patches/patch-tools_lldb-mi_MICmnBase.cpp
      13 patches/patch-tools_lldb-mi_MICmnBase.h
      13 patches/patch-tools_lldb-mi_MIDriver.cpp
      13 patches/patch-tools_lldb-mi_MIUtilString.cpp
      13 patches/patch-tools_lldb-mi_MIUtilString.h
      27 patches/patch-tools_lldb-server_CMakeLists.txt
    3539 total

1. Native Process NetBSD Plugin

I've created the initial code for the Native Process NetBSD Plugin.

process resume support (PT_CONTINUE)
process step support (PT_STEP) - currently not fully functional
functional callback monitor
functional process launch operation
appropriate ptrace(2) wrapper with logging capabilities
initial NativeThreadNetBSD code
setup for EVENT_MASK of a traced process (we monitor thread events)
preliminary code for other functions, like attach to a process and reading/writing memory of a tracee

2. The MonitorCallback function

The MonitorCallback function supports now the following events:

process exit and retrieve status code
software breakpoint (TRAP_BRKPT)
single step (TRAP_TRACE)
process image swich trap (TRAP_EXEC)
child traps (TRAP_CHLD) - currently detectable but disabled
thread trap (TRAP_LWP)
hardware assisted watchpoint trap (TRAP_HWWPT)

3. Other LLDB code, out of the NativeProcessNetBSD Plugin

During this work segment I've completed the following tasks:

created initial Native Process Plugin for NetBSD with remote debugging facilities, with skeleton copied from Linux but almost every function used so far had to be rewritten for NetBSD.
attached the Native Process Plugin for NetBSD to the build infrastructure.
disabled existing code for retrieving and updating Thread Name in NatBSD host plugin as currently not compatible with the LLDB codebase.
added NetBSD Process Launcher.
support proper native NetBSD Thread ID in Host::GetCurrentThreadID.
upgraded Platform NetBSD Plugin for new remote debugging plugin.

4. Automated LLDB Test Results Summary

The number of passing tests increased by 45% between devel/lldb 3.9.1 and lldb-netbsd 2017-01-21.

The above graphs renders test results for:

lldb 3.9.1 - the recent version from pkgsrc without newly developed patches
lldb-nbsd 2016-12-22 - the first startable version with the NativeProcessNetBSD Plugin
lldb-nbsd 2016-01-18 - the first version with functional MonitorCallback
lldb-nbsd 2016-01-21 - the first version with functional PT_CONTINUE and partially functional PT_STEP

Example LLDB sessions

It's demo time!

Breakpoint interrupt

In this example, I'm calling a hello world application that is triggering a software breakpoint. It's implemented by embedding int3 call on amd64. The debugger is capable of catching this and resuming till correct process termination.

$ lldb ./int3 
(lldb) target create "./int3"
Current executable set to './int3' (x86_64).
(lldb) r
Hello world!
Process 29578 launched: './int3' (x86_64)
Process 29578 stopped
* thread #1, stop reason = signal SIGTRAP
    frame #0:
(lldb) c
Process 29578 resuming
Process 29578 exited with status = 0 (0x00000000)
(lldb)

Thread monitor interrupt

In this example we set trap on thread events - creation and termination. The executed program incepts a thread and terminates afterwards, this is caught by the MonitorCallback with appropriate thread list update. After the end, program terminates correctly and passes proper exit status to the debugger.

$ lldb ./lwp_create
(lldb) target create "./lwp_create"
Current executable set to './lwp_create' (x86_64).
(lldb) r
Process 27331 launched: './lwp_create' (x86_64)
Hello world!
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
  thread #2, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
(lldb) thread list 
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
  thread #2: tid = 0x0002, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
Process 27331 stopped
* thread #1, stop reason = SIGTRAP has been caught with Process LWP Trap type
    frame #0:
(lldb) thread list
Process 27331 stopped
* thread #1: tid = 0x0001, stop reason = SIGTRAP has been caught with Process LWP Trap type
(lldb) c
Process 27331 resuming
It works
Process 27331 exited with status = 0 (0x00000000)
(lldb)

Plan for the next milestone

I've listed the following goals for the next milestone.

fix conflict with system-wide py-six
add support for auxv read operation
switch resolution of pid -> path to executable from /proc to sysctl(7)
recognize Real-Time Signals (SIGRTMIN-SIGRTMAX)
upstream !NetBSDProcessPlugin code
switch std::call_once to llvm::call_once
add new ptrace(2) interface to lock and unlock threads from execution
switch the current PT_WATCHPOINT interface to PT_GETDBREGS and PT_SETDBREGS

This work was sponsored by The NetBSD Foundation.

Previous report "Summary of the ptrace(2) project".

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can: