Feature #12475


want cyclic_move_here()

Added by Patrick Mooney about 2 years ago. Updated about 2 years ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


Upstreaming OS-6633 for bhyve:

When they are initially created, cyclics are generally placed on the cpu which was running the thread which allocated them. Reprogramming operations for the cyclic result in cross-call-like behavior when performed from other CPUs. This is fine for many cases, but certain applications may wish to localize that cyclic for better reprogramming performance. The cyclic_bind interface is a potential choice, but bound cyclics are not shuffled off a CPU, should it be offlined. To solve this problem, we propose adding a cyclic_move_here() interface to move a specific cyclic onto the CPU running the calling thread. Initially, this will be limited to cyclics which are not bound (to a CPU or partition).

Actions #1

Updated by Patrick Mooney about 2 years ago

This change was necessitated by bhyve's use of cyclics for timers in emulated devices. In particular the APIC timer present on every vCPU undergoes heavy reprogramming while a typical guest is under load. Prior to cyclic_move_here() being available, this meant many cbe_xcall() invocations to reprogram cyclics which were on a different physical CPU than where the vCPU was currently running (since the vAPIC exit processing for the timer reprogramming would happen there). Since cyclics use that bespoke cbe_xcall() method, rather than traditional OS cross-calls, they aren't visible as xcalls under mpstat, but would appear as a high number of IPIs on the system. With this in place, and wired appropriately into bhyve, IPIs were significantly reduced, as was contention on locks used for cross-CPU cyclic reprogramming.

Actions #2

Updated by Electric Monk about 2 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit d0a94a58eef5c83b34de379fe636b84a98ec79a0

commit  d0a94a58eef5c83b34de379fe636b84a98ec79a0
Author: Patrick Mooney <>
Date:   2020-04-16T16:44:53.000Z

    12475 want cyclic_move_here()
    Reviewed by: Bryan Cantrill <>
    Reviewed by: Jerry Jelinek <>
    Reviewed by: Toomas Soome <>
    Reviewed by: Yuri Pankov <>
    Approved by: Robert Mustacchi <>


Also available in: Atom PDF