Feature #13833
closed
want bhyve memory reservoir
100%
Description
On hosts which are primarily intended to run bhyve VMs, it would be advantageous to set aside memory in a reservoir specifically meant for bhyve guest memory allocations. This would allow bhyve to quickly and confidently allocate memory for instances without any chance of the ARC (or other large consumers) to soak up free pages, forcing potentially lengthy reaping activity. Use of such a reservoir could be explicitly opt-in, where memory was added to the reservoir via a CLI utility, and bhyve instances were configured to allocate from it when starting. Alternatively, on systems which did not wish to dedicate memory to bhyve instances (particularly when they were not running), bhyve could make "transient" allocations which function similar to things today (where memory is scrounged from the freelist, and reaping from the ARC and kmem).
Related issues
Updated by Patrick Mooney 3 months ago
- Related to Bug #13896: bhyve VM interfaces should be better fit added
Updated by Michael Zeller 2 months ago
You can find a test platform for SmartOS here (manta currently has an issue so it's stashed on my webserver):
http://lightsandshapes.com/platform-20210713T161045Z.tgz
If you wish to see the changes or build them yourself you can find a branch with the commits here:
https://github.com/joyent/illumos-joyent/tree/reservoir-testing
Updated by Michael Zeller 2 months ago
The latest bits are now at https://us-east.manta.joyent.com/mikezeller/public/platform-20210713T161045Z.tgz
The branch has also been updated.
Updated by Patrick Mooney about 2 months ago
Testing for this was a combination of isolated experimentation and end-to-end usage:
- Adding/removing chunks of memory from the reservoir with rsrvrctl
- Starting/stopping VMs which allocate from the reservoir
- Starting/stopping VMs which perform transient allocations (the default behavior for non-reservoir-aware bhyve)
- Bringing the host into situations of memory pressure (while doing the above)
- After other testing was complete, dumping the machine and checking ::findleaks
These tests were run primarily on AMD hardware, although it was checked on an Intel machine as well.
Updated by Patrick Mooney about 2 months ago
It should be noted that the size limit imposed by the reservoir system (shown in rsrvrctl -q as Size limit KiB:) is not adequate for preventing a machine administrator from attempting to add so much memory to the reservoir such that memory pressure makes the system borderline unresponsive. It has been a challenge to find a static calculation that would cover all possible cases, since kernel memory consumption can vary wildly between different hardware configurations. The existing limit was chosen as an attempt to at least keep us out of needfree without being a burden on machines with large amounts of DRAM.
Updated by Electric Monk about 2 months ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit b57f5d3e6a2df8d435e606797cf3934811848343
commit b57f5d3e6a2df8d435e606797cf3934811848343
Author: Patrick Mooney <pmooney@pfmooney.com>
Date: 2021-07-27T19:26:22.000Z
13833 want bhyve memory reservoir
13822 bhyve memory should exert memory pressure
13834 want extensible page_resv
13821 vmmctl ioctls should have more structure
Reviewed by: Andy Fiddaman <andy@omnios.org>
Reviewed by: Jason King <jason.brian.king@gmail.com>
Reviewed by: Dan Cross <cross@oxidecomputer.com>
Reviewed by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
Reviewed by: Mike Zeller <mike.zeller@joyent.com>
Approved by: Dan McDonald <danmcd@joyent.com>