Feature #12475
closed
want cyclic_move_here()
100%
Description
Upstreaming OS-6633 for bhyve:
When they are initially created, cyclics are generally placed on the cpu which was running the thread which allocated them. Reprogramming operations for the cyclic result in cross-call-like behavior when performed from other CPUs. This is fine for many cases, but certain applications may wish to localize that cyclic for better reprogramming performance. The cyclic_bind interface is a potential choice, but bound cyclics are not shuffled off a CPU, should it be offlined. To solve this problem, we propose adding a cyclic_move_here() interface to move a specific cyclic onto the CPU running the calling thread. Initially, this will be limited to cyclics which are not bound (to a CPU or partition).
Updated by Patrick Mooney about 2 years ago
This change was necessitated by bhyve's use of cyclics for timers in emulated devices. In particular the APIC timer present on every vCPU undergoes heavy reprogramming while a typical guest is under load. Prior to cyclic_move_here() being available, this meant many cbe_xcall() invocations to reprogram cyclics which were on a different physical CPU than where the vCPU was currently running (since the vAPIC exit processing for the timer reprogramming would happen there). Since cyclics use that bespoke cbe_xcall() method, rather than traditional OS cross-calls, they aren't visible as xcalls under mpstat, but would appear as a high number of IPIs on the system. With this in place, and wired appropriately into bhyve, IPIs were significantly reduced, as was contention on locks used for cross-CPU cyclic reprogramming.
Updated by Electric Monk about 2 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit d0a94a58eef5c83b34de379fe636b84a98ec79a0
commit d0a94a58eef5c83b34de379fe636b84a98ec79a0
Author: Patrick Mooney <pmooney@pfmooney.com>
Date: 2020-04-16T16:44:53.000Z
12475 want cyclic_move_here()
Reviewed by: Bryan Cantrill <bryan@joyent.com>
Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Reviewed by: Toomas Soome <tsoome@me.com>
Reviewed by: Yuri Pankov <ypankov@fastmail.com>
Approved by: Robert Mustacchi <rm@fingolfin.org>