posix thread priority functions are broken for TS class
For TS threads, pthread_setschedprio(3c) and pthread_setschedparam(3c) (and thr_setprio(3c) all (due to common implementation) change not only the priority of a thread to the given value, but also set uprilim to the same value (see usr/src/lib/libc/port/rt/sched.c:set_priority)
This has the surprising effect of only ever allowing a thread to lower it's priority, never raise it (even to a previous value). There is nothing I can find documented that suggests this is intended behavior, none of the other scheduling classes (excluding RT as that's a different beast altogether) do this, and if you use priocntl(2) directly instead, you can raise and lower a thread's priority (within it's allowed range) without issue.
The simple fix is to remove the line 'tsp->ts_uprilim = prio;' from set_priority() in sched.c
Updated by John Levon about 2 months ago
Looking at this again, this is happening because we don't actually know ->ts_uprilim at set_priority(), so we can't trivially do the check I suggested. So that most likely explains the current code: it needs to fully fill out the parms_t struct, without knowing the values already.