This has sat here for a long time pending careful review, because the logic is
not easy to follow. Fortunately I think I understand it now.
The described race is pretty much accurate. When a thread called
RtlExitUserThread() in 1.7.38, it first decremented nb_threads. If it was the
last thread, it would call exit() with the thread exit status; if not, it would
mask off signals [the order here is important] and then call pthread_exit() with
the status.
When a thread called RtlExitUserProcess(), this happened:
* The caller does a terminate_process() request to the server, which sends
SIGQUIT to every thread *but* the caller.
* The SIGQUIT handler calls terminate_thread() with a zero status.
terminate_thread() masks off signals, then decrements nb_threads. If the
aborting thread is the last thread, it would call _exit(), otherwise, it'd
again just pthread_exit().
* Finally, the original thread would call exit(), with the intended status
code.
[All of the intermediate function calls and helpers are skipped for brevity and
clarity].
The problem happens if both of these happen at the same time in different
threads. In this case the RtlExitUserThread() thread could decrement nb_threads,
then get interrupted by SIGQUIT and decrement nb_threads again. The end result
is that, instead of the RtlExitUserProcess() thread exiting with the intended
status, instead one of the SIGQUIT threads will be the "last" thread, and exit
with the status that SIGQUIT uses, which is zero as described.
A more serious race than this can be constructed if a thread is terminated by
another thread while already exiting. In this case nb_threads would be executed
twice, but the consequence would be that the *penultimate* thread to exit,
later, would end up killing the process, since it thinks it's the ultimate
thread.
2334f4e64582a518e4d5a7627472a0d817b147ef changed this. Now a thread calling
RtlExitUserThread() does not decrement nb_threads, but instead asks the server
if it's the last thread, and if so exit the whole process [at the time via
exit(); later via RtlExitUserProcess().] If not the last thread, the threads
mask off signals and then call pthread_exit() as before.
This avoided the race, but added a different one, essentially the opposite
problem: if two threads exit cleanly at the same time, neither one of them will
think they're the last thread, then both will exit without calling exit().
Apparently (from IRC logs) this would leave the thread in a weird state where
it'd still be running somehow, although it's not really clear how.
In any case, this problem was fixed by fac1aabbef3753afc53a4ea4f933b3d0516fd302
upstream. Now if two threads call NtTerminateThread() on themselves at the same
time, they really will exit cleanly and one will terminate the process.
Critically, this is now safe from the original race, because decrementing
nb_threads is done after masking off signals.
e1966ac26e