Summary of the ptrace(2) project

In short, we are already in a good state with the existing ptrace(2) interfaces, as most necessary functions in LLDB are representable by existing NetBSD specific interfaces. We can fully implement core LLDB functionality without further extensions in ptrace(2). During this project dozen of bugs were investigated & fixed and several hundreds of ATF tests added. The major addition is newly added support for hardware assisted watchpoints API for ptrace(2) on amd64 and preliminary code for i386 and XEN.

What has been done

1. Verified basic ptrace(2) functionality:

• debugger cannot attach to PID0 (as root and user)
• debugger cannot attach to PID1 (as user)
• debugger cannot attach to PID1 with sercurelevel >= 1 (as root and user)
• debugger cannot attach to self
• debugger cannot attach to another process unless the process's root directory is at or below the tracing process's root

2. Verified the full matrix of combinations of wait(2) and ptrace(2) in the following test-cases

• tracee can emit PT_TRACE_ME for parent and raise SIGSTOP followed by _exit(2)
• tracee can emit PT_TRACE_ME for parent and raise SIGSTOP followed by _exit(2), with perent sending SIGINT and catching this singal only once with a signal handler and without termination of tracee
• tracee can emit PT_TRACE_ME for parent and raise SIGSTOP, with perent sending SIGINT and terminating the child without signal handler
• tracee can emit PT_TRACE_ME for parent and raise SIGSTOP, with perent sending SIGINT and terminating the child without signal handler
• tracee can emit PT_TRACE_ME for parent and raise SIGCONT, and parent reports it as process stopped
• assert that tracer sees process termination earlier than the parent
• assert that any tracer sees process termination earlier than its parent
• assert that tracer parent can PT_ATTACH to its child
• assert that tracer child can PT_ATTACH to its parent
• assert that tracer sees its parent when attached to tracer (check getppid(2))
• assert that tracer sees its parent when attached to tracer (check sysctl(7) and struct kinfo_proc2)
• assert that tracer sees its parent when attached to tracer (check /proc/curproc/status 3rd column)
• verify that empty EVENT_MASK is preserved
• verify that PTRACE_FORK in EVENT_MASK is preserved
• verify that fork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_FORK
• verify that fork(2) is not intercepted by ptrace(2) with empty EVENT_MASK
• verify that vfork(2) is intercepted by ptrace(2) with EVENT_MASK set to PTRACE_VFORK [currently EVENT_VFORK not implemented]
• verify that vfork(2) is not intercepted by ptrace(2) with empty EVENT_MASK [currently failing as EVENT_VFORK not implemented]
• verify PT_IO with PIOD_READ_D and len = sizeof(uint8_t)
• verify PT_IO with PIOD_READ_D and len = sizeof(uint16_t)
• verify PT_IO with PIOD_READ_D and len = sizeof(uint32_t)
• verify PT_IO with PIOD_READ_D and len = sizeof(uint64_t)
• verify PT_IO with PIOD_WRITE_D and len = sizeof(uint8_t)
• verify PT_IO with PIOD_WRITE_D and len = sizeof(uint16_t)
• verify PT_IO with PIOD_WRITE_D and len = sizeof(uint32_t)
• verify PT_IO with PIOD_WRITE_D and len = sizeof(uint64_t)
• verify PT_READ_D called once
• verify PT_READ_D called twice
• verify PT_READ_D called three times
• verify PT_READ_D called four times
• verify PT_WRITE_D called once
• verify PT_WRITE_D called twice
• verify PT_WRITE_D called three times
• verify PT_WRITE_D called four times
• verify PT_IO with PIOD_READ_D and PIOD_WRITE_D handshake
• verify PT_IO with PIOD_WRITE_D and PIOD_READ_D handshake
• verify PT_READ_D with PT_WRITE_D handshake
• verify PT_WRITE_D with PT_READ_D handshake
• verify PT_IO with PIOD_READ_I and len = sizeof(uint8_t)
• verify PT_IO with PIOD_READ_I and len = sizeof(uint16_t)
• verify PT_IO with PIOD_READ_I and len = sizeof(uint32_t)
• verify PT_IO with PIOD_READ_I and len = sizeof(uint64_t)
• verify PT_READ_I called once
• verify PT_READ_I called twice
• verify PT_READ_I called three times
• verify PT_READ_I called four times
• verify plain PT_GETREGS call without further steps
• verify plain PT_GETREGS call and retrieve PC
• verify plain PT_GETREGS call and retrieve SP
• verify plain PT_GETREGS call and retrieve INTRV
• verify PT_GETREGS and PT_SETREGS calls without changing regs
• verify plain PT_GETFPREGS call without further steps
• verify PT_GETFPREGS and PT_SETFPREGS calls without changing regs
• verify single PT_STEP call
• verify PT_STEP called twice
• verify PT_STEP called three times
• verify PT_STEP called four times
• verify that PT_CONTINUE with SIGKILL terminates child
• verify that PT_KILL terminates child
• verify basic LWPINFO call for single thread (PT_TRACE_ME)
• verify basic LWPINFO call for single thread (PT_ATTACH from tracer)

3. Documentation of ptrace(2)

• documented PT_SET_EVENT_MASK, PT_GET_EVENT_MASK and PT_GET_PROCESS_STATE
• updated and fixed documentation of PT_DUMPCORE
• documented PT_GETXMMREGS and PT_SETXMMREGS (i386 port specific)
• documented PT_GETVECREGS and PT_SETVECREGS (ppc ports specific)
• other tweaks and cleanups in the documentation

4. exect(3) - execve(2) wrapper with tracing capabilities

Researched its usability and added ATF test, it's close to be marked for removal - it's marked as broken as it is on all ports... this call was inherited from BSD4.2 (VAX) and was never useful since the inception as it is enabling singlestepping before calling execve(2) and tracing libc calls before switching to new process image.

5. pthread_dbg(3) - POSIX threads debugging library documentation

• added documentation for the library in man-page
• upstreamed to the mandoc project to recognize the pthread_dbg(3) library
• document td_close(3) close connection to a threaded process
• document td_map_pth2thr(3) convert a pthread_t to a thread handle
• document td_open(3) make connection to a threaded process
• document td_thr_getname(3) get the user-assigned name of a thread
• document td_thr_info(3) get information on a thread
• document td_thr_iter(3) iterate over the threads in the process

6. pthread_dbg(3) - pthread debug library - t_dummy tests

• assert that dummy lookup functions stops td_open(3)
• assert that td_open(3) for basic proc_{read,write,lookup} works
• asserts that calling td_open(3) twice for the same process fails

7. pthread_dbg(3) - pthread debug library - test of features

• assert that td_thr_iter(3) call without extra logic works
• assert that td_thr_iter(3) call is executed for each thread once
• assert that for each td_thr_iter(3) call td_thr_info(3) is valid
• assert that for each td_thr_iter(3) call td_thr_getname(3) is valid
• assert that td_thr_getname(3) handles shorter buffer parameter and the result is properly truncated
• assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- and assert earlier that td_thr_iter(3) call is valid
• assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- and assert later that td_thr_iter() call is valid
• assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- compare thread's name of pthread_t and td_thread_t
• assert that pthread_t can be translated with td_map_pth2thr(3) to td_thread_t -- assert that thread is in the TD_STATE_RUNNING state

8. pthread_dbg(3) - pthread debug library - code fixes

• fix pt_magic (part of pthread_t) read in td_thr_info(3)
• correct pt_magic reads in td_thr_{getname,suspend,resume}(3)
• fix pt_magic read in td_map_pth2thr(3)
• always set trailing '\0' in td_thr_getname(3) to compose valid ASCIIZ string
• kill SA thread states (TD_STATE_*) in pthread_dbg and add TD_STATE_DEAD
• obsolete thread_type in td_thread_info_st in pthread_dbg.h

This library was designed for Scheduler Activation, and this feature was removed in NetBSD 5.0.

9. wait(2) family tests

• test that wait6(2) handled stopped/continued process loop
• test whether wait(2)-family of functions return error and set ECHILD for lack of children
• test whether wait(2)-family of functions return error and set ECHILD for lack of children, with WNOHANG option verifying that error is still signaled and errno set

10. Debug Registers assisted watchpoints:

• fix rdr6() function on amd64
• add accessors for available x86 Debug Registers (DR0-DR3, DR6, DR7)
• switch x86 CPU Debug Register types from vaddr_t to register_t
• torn down KSTACK_CHECK_DR0, i386-only feature to detect stack overflow

12. i386 port tests

• call PT_GETREGS and iterate over General Purpose registers

13. amd64 port tests

• call PT_GETREGS and iterate over General Purpose registers
• call PT_COUNT_WATCHPOINTS and assert four available watchpoints
• call PT_COUNT_WATCHPOINTS and assert four available watchpoints, verify that we can read these watchpoints
• call PT_COUNT_WATCHPOINTS and assert four available watchpoints, verify that we can read and write unmodified these watchpoints
• call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 0
• call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 1
• call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 2
• call PT_COUNT_WATCHPOINTS and test code trap with watchpoint 3
• call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 0
• call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 1
• call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 2
• call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 3
• call PT_COUNT_WATCHPOINTS and test write trap with watchpoint 0
• call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 0 on data read
• call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 1 on data read
• call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 2 on data read
• call PT_COUNT_WATCHPOINTS and test rw trap with watchpoint 3 on data read

14. Other minor improvements

• document in wtf(7) PCB process control block
• fix cpu_switchto(9) prototype in a comment in RICSV

15. Add support for hardware assisted watchpoints/breakpoints API in ptrace(2)

Add new ptrace(2) calls:

1. PT_COUNT_WATCHPOINTS - count the number of available hardware watchpoints
2. PT_READ_WATCHPOINT - read struct ptrace_watchpoint from the kernel state
3. PT_WRITE_WATCHPOINT - write new struct ptrace_watchpoint state, this includes enabling and disabling watchpoints

The ptrace_watchpoint structure contains MI and MD parts:

typedef struct ptrace_watchpoint {
	int		pw_index;	/* HW Watchpoint ID (count from 0) */
	lwpid_t		pw_lwpid;	/* LWP described */
	struct mdpw	pw_md;		/* MD fields */
} ptrace_watchpoint_t;

For example amd64 defines MD as follows:

struct mdpw {
	void	*md_address;
	int	 md_condition;
	int	 md_length;
};

These calls are protected with the __HAVE_PTRACE_WATCHPOINTS guard.

Tested on amd64, initial support added for i386 and XEN.

16. Reported bugs:

• PR kern/51596 (ptrace(2): raising SIGCONT in a child results with WIFCONTINUED and WIFSTOPPED true in the parent)
• PR kern/51600 (Tracer must detect zombie before child's parent)
• PR standards/51603 WIFCONTINUED()=true always implies WIFSTOPPED()=true
• PR standards/51606 wait(2) with WNOHANG does not return errno ECHILD for lack of children
• PR kern/51621: PT_ATTACH from a parent is unreliable [false positive]
• PR kern/51624: Tracee process cannot see its appropriate parent when debugged by a tracer
• PR kern/51630 ptrace(2) command PT_SET_EVENT_MASK: option PTRACE_VFORK unsupported
• PR lib/51633 tests/lib/libpthread_dbg/t_dummy unreliable [false positive]
• PR lib/51635 td_thr_iter in seems broken [false positive]
• PR lib/51636: It's not possible to run under gdb(1) programs using the pthread_dbg library
• PR kern/51649: PRIxREGISTER and PTRACE_REG_* mismatch
• PR kern/51685 (ptrace(2): Signal does not set PL_EVENT_SIGNAL in (struct ptrace_lwpinfo.)pl_event)
• PR port-amd64/51700 exect(3) misdesigned and hangs

.... and several critical ones not reported and fixed directly by the NetBSD team.

Credit for Christos Zoulas and K. Robert Elz for helping with the mentioned bugs.

17. Added doc/TODO.ptrace entries

• verify ppid of core dump generated with PT_DUMPCORE it must point to the real parent, not tracer
• adapt OpenBSD regress test (regress/sys/ptrace/ptrace.c) for the ATF context
• add new ptrace(2) calls to lock (suspend) and unlock LWP within a process
• add PT_DUMPCORE tests in the ATF framework
• add ATF tests for PT_WRITE_I and PIOD_WRITE_I - test mprotect restrictions
• add ATF tests for PIOD_READ_AUXV
• add tests for the procfs interface covering all functions available on the same level as ptrace(2)
• add support for PT_STEP, PT_GETREGS, PT_SETREGS, PT_GETFPREGS, PT_SETFPREGS in all ports
• integrate all ptrace(2) features in gdb
• add ptrace(2) NetBSD support in LLDB
• add support for detecting equivalent events to PTRACE_O_TRACEEXEC, PTRACE_O_TRACECLONE, PTRACE_O_TRACEEXIT from Linux
• exect(3) rething or remove -- maybe PT_TRACE_ME + PTRACE_O_TRACEEXEC?
• refactor pthread_dbg(3) to only query private pthread_t data, otherwise it duplicates ptrace(2) interface and cannot cover all types of threads

Features in ELF, DWARF, CTF, DTrace are out of scope of the above list.

Plan for the coming weeks

My initial goal is to copy the Linux Process Plugin and add minimal functional support for NetBSD in LLDB. It will be followed with running and passing some tests from the lldb-server test-suite.

This is a shift from the original plan about porting FreeBSD Process Plugin to NetBSD, as the FreeBSD one is lacking remote debugging support and it needs to be redone from scratch.

I'm going to fork wip/lldb-git (as for git snapshot from 16 Dec 2016) for the purpose of this task to wip/lldb-netbsd and work there.

Next steps after finishing this task are to sync up with Pavel from the LLDB team after New Year. The NetBSD Process Plugin will be used as a reference to create new Common Process Plugin shared between Linux and (Net)BSD.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate [6 comments]