illumos gate

We have additional, non-open-source code which makes heavy usage of the fpu while in the kernel. While stress testing this code we can hit this panic after 12-36 hours of constant testing.
There are various race bugs in kernel_fpu_begin and kernel_fpu_end which can cause a context switch at the wrong time and leave the thread's pcb fpu state incorrect.

During debugging and fixing this, I took the opportunity to also cleanup some of the code in kernel_fpu_end and to improve the comments.

The most pernicious bug here was when the code would context switch after the fp_save in kernel_fpu_begin, but before the FPU_KERNEL flag was set. This meant that we would leave kernel_fpu_begin without the user-land FPU state being valid in the pcb. Thus, if we ever context switch off the new cpu and that cpu had previously had a system thread running on it which had used the kernel fpu, then that cpu would still have the TS bit set in %cr0 (from fpdisable in kernel_fpu_end and since the user-land restore handler would not have run to do CLTS), so then we would panic.

Of course this also implies we have corrupted the user-land fpu state, which is bad, although in practice the user-land threads we have mostly observed (but not always) are SMB threads which come into the kernel but never leave.

For testing this, we developed a stress load test which would reliably panic within 12 hours. After this fix we ran this load continuously for about 5 days without a panic.