Bug #11494
opensynchronous signals delivered with inappropriate context when using ITIMER_PROF
0%
Description
Some fault conditions experienced by user programs are reported via signals, directed to the LWP which induced the fault. The list of such signals includes at least SIGSEGV
, SIGBUS
, SIGILL
, SIGTRAP
, and SIGFPE
. When delivering a fault signal like this, we populate the siginfo_t
with information about the fault address, and potentially the program counter, at the moment of the fault. In addition, some software (not unreasonably) will inspect the ucontext_t
that we provide to the signal handler to collect more detailed information about the state of the program when it was interrupted by the signal. This is used by language runtimes like the JVM to convert a SIGSEGV
into an appropriate NullPointerException
, amongst other things.
When deciding which signal to deliver In issig_forreal()
, we use fsig()
twice: first, to check the thread-directed pending signal set; then, if there are no thread-directed signals, we check the process-directed signal set. In general, with a couple of exceptions, fsig()
starts with the lowest numbered signal (i.e., it will try SIGBUS
, 10, before SIGSEGV
, 11).
The synchronous fault signals are all thread-directed, so they are given priority over any signals from outside. We do not, to my knowledge, allow processes to signal an arbitrary thread in another process -- they use kill(2)
, which only generates process-directed signals. Faulting signals, therefore, are effectively delivered first. The context we preserve from the interrupted program will reflect the machine state when the fault occurred.
Unfortunately, there are conditions under which an exception made in fsig()
causes the incorrect behaviour. If SIGKILL
is present in the inspected signal set, it will be prioritised above any other signal. That makes sense, because if KILL
is asserted, the program is over before it gets to any other signal handling anyway. In addition to KILL
, though, we give second place to SIGPROF
(number 29!) before attending the rest of the signals in ascending order. When SIGPROF
is process-directed, this doesn't matter because it falls in line behind any of the thread-directed fault signals. When using setitimer(2)
with the ITIMER_PROF
timer, however, the kernel _sends a thread-directed SIGPROF
in clock_tick()
if the LWP is on-CPU when the tick occurs!
This thread-directed SIGPROF
will jump the queue to begin delivery prior to any fault signal that might have been tripped at around the time that the clock tick occurred. If the signal mask (sa_mask
) provided to sigaction(2)
for SIGPROF
did not mask other signals, as soon as we unmask before calling the user handler, we'll be dragged back into the kernel to deliver the fault signal. We're now in nested signal handling territory, so the context we provide with that faulting signal reflects an interrupted libc
signal handling function, not the user program.
A rough sequence of events that leads to the problem:
- user program asks for some unmapped memory
- page fault handler is triggered, determines the access is invalid, sets up a
SIGSEGV
for this LWP clock_tick()
sends a thread-directedSIGPROF
because we are presently on-CPU- on the way back to user mode we see there are signals so we determine which one to send first;
fsig()
chooses theSIGPROF
- we return to user mode, vectored to the
libc
signal handling machinery, with all signals blocked and a context that refers to the main program that tripped the fault libc
applies thesa_mask
for ourSIGPROF
handler, unblocking signals by making a_lwp_sigmask()
system call- on return from
_lwp_sigmask()
, we see there are signals andfsig()
tells usSIGSEGV
is up next - we are revectored to the
SIGSEGV
handler, with a context that refers tolibc
just after the_lwp_sigmask()
call - the user program's
SIGSEGV
handler flies off the rails becauseREG_PC
doesn't point to the program text that faulted in the first place
Note that there is a wholly separate delivery mechanism for use of setitimer(2)
with ITIMER_REALPROF
which will also generate SIGPROF
but not in this way, and does not appear to exhibit these same symptoms.
It seems reasonable not to prioritise SIGPROF
before the synchronous fault signals (which are otherwise all lower in number than 29). Even if fsig()
did nothing special with it, it will still be prioritised above signals from other processes by virtue of being thread-directed -- it just won't interfere with the correct delivery of context about SIGSEGV
, SIGILL
, etc.
Updated by Joshua M. Clulow almost 3 years ago
There is some dubious language in comments in the JDK which seems like it probably addresses this condition. Here, when attempting to resolve the ucontext_t
to use, we'll walk up the chain just one link (not an arbitrary number of links, as would be necessary to deal with potential nesting of async signals!):
// We will only follow one level of uc_link since there are libthread // issues with ucontext linking and it is better to be safe and just // let caller retry later. const ucontext_t* os::Solaris::get_valid_uc_in_signal_handler(Thread *thread, const ucontext_t *uc) { const ucontext_t *retuc = NULL; if (uc != NULL) { if (uc->uc_link == NULL) { // cannot validate without uc_link so accept current ucontext retuc = uc; } else if (os::Solaris::valid_ucontext(thread, uc, uc->uc_link)) { // first ucontext is valid so try the next one uc = uc->uc_link; if (uc->uc_link == NULL) { // cannot validate without uc_link so accept current ucontext retuc = uc; } else if (os::Solaris::valid_ucontext(thread, uc, uc->uc_link)) { // the ucontext one level down is also valid so return it retuc = uc; } } } return retuc; }
(Note mention of SIGPROF
for profiling appears elsewhere in this file)
Updated by Joshua M. Clulow almost 3 years ago
I believe this is also the cause of this bug in SBCL: https://bugs.launchpad.net/sbcl/+bug/1248181
Updated by Ludovic Orban almost 3 years ago
I'm not sure if that can help, but for your reference, Cliff Click (the original Sun Lead Architect of the JVM) briefly mentioned what looks like this problem in one of his talks: https://www.youtube.com/watch?v=vzzABBxo44g&t=14m31s
Updated by Joshua M. Clulow almost 3 years ago
Thanks for the reference! His description definitely sounds like this kind of problem.