February 10, 2026 · War Story
The TLS Bug That Took Three Days to Find
This is the story of a bug that produced zero error messages, no crashes at the point of failure, and symptoms that appeared hundreds of instructions later in completely unrelated code.
The Symptom
Dynamically-linked FreeBSD binaries would start executing, load the dynamic linker (ld-elf.so.1), resolve symbols, and then segfault deep inside libthr (FreeBSD's threading library) during initialization. Static binaries worked fine.
The Red Herrings
First suspicion: the ELF loader was setting up the auxiliary vector wrong. Hours of checking AT_BASE, AT_PHDR, AT_ENTRY — all correct.
Second suspicion: shared library loading was corrupting memory. Traced every mmap call, verified every protection flag. All correct.
Third suspicion: signal handling was interfering. Disabled all signal translation. Same crash.
The Actual Bug
FreeBSD's libthr uses the sysarch(AMD64_SET_FSBASE) syscall to set the thread pointer for Thread Local Storage. Our emulation of this was working correctly — it used PTRACE_ARCH_PRCTL to set the FS base register in the child process.
The problem: we were setting it too early.
The dynamic linker sets up TLS during its initialization phase. But after TLS setup, the linker continues initializing the BSS (uninitialized data) sections of loaded libraries. This zeroing pass was overwriting the memory region where the thread pointer had been stored.
The FS base register still pointed to the correct address. But the data at that address had been zeroed by BSS initialization. When libthr later dereferenced the thread pointer, it found zeros instead of the thread control block, and segfaulted.
The Fix
The fix was to defer the FS base write. Instead of immediately executing the sysarch emulation when the syscall is intercepted, we record the requested base address and apply it at the right point in the initialization sequence — after BSS zeroing is complete but before any threading code runs.
This required understanding the exact sequence of operations in FreeBSD's rtld (runtime linker) initialization, which meant reading the FreeBSD source code for libexec/rtld-elf/rtld.c to understand when BSS zeroing happens relative to TLS setup.
Lesson
In syscall translation, timing matters as much as correctness. A perfectly correct emulation executed at the wrong point in a process's lifecycle is just as broken as a wrong one.