illumos gate

Upstreaming a select portion of OS-5845 for bhyve:

As part of the work on LX branded zones in illumos-joyent, we wanted the ability to run kernel-only threads associated with a process. This was initially used for emulated Linux AIO in-kernel, rather than via the userspace library.

When we were working to improve viona for bhyve, the convenience of instantiating those kernel only threads in the bhyve process for virtio ring processing was convenient. That specific functionality is required before the modern viona implementation is upstreamed.

The gist of this approach is to create an lwp in the process. This thread has a special flag set (TP_KTHREAD) which both excepts it from selection in prchoose() and differentiates it in the kernel heap, should it matter for inspection of a crash dump. In most cases (for viona, at least) the worker thread will have all signals blocked (although paying heed to the containing process for SEXITING) and execute an function which will not make a trip to userspace.

An example of how one might create such a thread:

k_sigset_t hold_set;
proc_t *p = curproc;
kthread_t *t;
klwp_t *lwp;

sigfillset(&hold_set);
lwp = lwp_create(WORKER_FUNC, (void *)WORKER_ARG, 0, p, TS_STOPPED,
        minclsyspri - 1, &hold_set, curthread->t_cid, 0);
if (lwp == NULL) {
        // failure path
        return;
}

t = lwptot(lwp);
mutex_enter(&p->p_lock);
t->t_proc_flag = (t->t_proc_flag & ~TP_HOLDLWP) | TP_KTHREAD;
lwp_create_done(t);
mutex_exit(&p->p_lock);

The thread will live and run inside the process from which it was started. It's the responsibility of that thread to check for status changes (like SEXITING) in the process which would precipitate its own exit.