Project

General

Profile

Feature #12675

want kthread worker interface

Added by Patrick Mooney 9 months ago. Updated 8 months ago.

Status:
Closed
Priority:
Normal
Category:
kernel
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
bhyve
Gerrit CR:

Description

Upstreaming a select portion of OS-5845 for bhyve:

As part of the work on LX branded zones in illumos-joyent, we wanted the ability to run kernel-only threads associated with a process. This was initially used for emulated Linux AIO in-kernel, rather than via the userspace library.

When we were working to improve viona for bhyve, the convenience of instantiating those kernel only threads in the bhyve process for virtio ring processing was convenient. That specific functionality is required before the modern viona implementation is upstreamed.

The gist of this approach is to create an lwp in the process. This thread has a special flag set (TP_KTHREAD) which both excepts it from selection in prchoose() and differentiates it in the kernel heap, should it matter for inspection of a crash dump. In most cases (for viona, at least) the worker thread will have all signals blocked (although paying heed to the containing process for SEXITING) and execute an function which will not make a trip to userspace.

An example of how one might create such a thread:

k_sigset_t hold_set;
proc_t *p = curproc;
kthread_t *t;
klwp_t *lwp;

sigfillset(&hold_set);
lwp = lwp_create(WORKER_FUNC, (void *)WORKER_ARG, 0, p, TS_STOPPED,
        minclsyspri - 1, &hold_set, curthread->t_cid, 0);
if (lwp == NULL) {
        // failure path
        return;
}

t = lwptot(lwp);
mutex_enter(&p->p_lock);
t->t_proc_flag = (t->t_proc_flag & ~TP_HOLDLWP) | TP_KTHREAD;
lwp_create_done(t);
mutex_exit(&p->p_lock);

The thread will live and run inside the process from which it was started. It's the responsibility of that thread to check for status changes (like SEXITING) in the process which would precipitate its own exit.


Related issues

Related to illumos gate - Feature #12730: document kthread worker interfaceNew

Actions
#1

Updated by Patrick Mooney 9 months ago

  • Description updated (diff)
#2

Updated by Patrick Mooney 9 months ago

  • Description updated (diff)
#3

Updated by Patrick Mooney 9 months ago

The portion of this required for bhyve/viona upstreaming is only the TP_KTHREAD addition (and its consuming logic in prchoose()).

#4

Updated by Patrick Mooney 9 months ago

  • Description updated (diff)
#5

Updated by Patrick Mooney 8 months ago

This has been in use by viona for bhyve (and LX) on SmartOS and OmniOSce for over a year now.

#6

Updated by Patrick Mooney 8 months ago

On a system with this patch (and the subsequent bhyve patches so that a consumer, viona, would be present) I tried some of the process-related actions which Jerry tested when he originally did the work. As expected, the kernel worker threads are absent from pstack and from the core file generated by gcore. They do appear in /proc and prstat with the expected names. Stopping and starting the process with pstop/prun went without a hitch as well.

#7

Updated by Patrick Mooney 8 months ago

#8

Updated by Electric Monk 8 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 4c819f48a094cdf29d044bcea57733ebac5f688f

commit  4c819f48a094cdf29d044bcea57733ebac5f688f
Author: Jerry Jelinek <jerry.jelinek@joyent.com>
Date:   2020-05-14T01:57:00.000Z

    12675 want kthread worker interface
    Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
    Reviewed by: Bryan Cantrill <bryan@joyent.com>
    Reviewed by: Gordon Ross <gordon.w.ross@gmail.com>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Also available in: Atom PDF