illumos gate

If more than one DTrace process has enablings for a set of probes and those processes instrument pid:::entry prior to return, DTrace will mis-detect the presence of another debugger due to the fasttrap at offset 0 (the entry probe) and refuse to create the return probe.

 libdtrace DEBUG: creating probe pid17625:date:main:return
 libdtrace DEBUG: error at 0 (assuming jump table)

The "assuming jump table" message comes from source:usr/src/lib/libdtrace/i386/dt_isadep.c#L97, though we did not hit a jump table but instead a completely invalid instruction, the invalid instruction is the -1 return from dt_instr_size source:usr/src/lib/libdtrace/i386/dt_isadep.c#L458. Further investigation shows that dtrace_disx86 is succeeding and we're returning -1 because we see the trap.

dtrace_disx86 attempts to get the original instruction when encountering a trap by ioctling the kernel to request it in dt_getbyte source:usr/src/lib/libdtrace/i386/dt_isadep.c#L428

After a brief email conversation with ahl, he discovered that this is likely because in dt_instr_size we examine the original instruction when making the FASTTRAP_INSTR comparison, rather than the instruction in the structure which dtrace_disx86 returns.