Feature #13833


want bhyve memory reservoir

Added by Patrick Mooney over 2 years ago. Updated over 2 years ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:
External Bug:


On hosts which are primarily intended to run bhyve VMs, it would be advantageous to set aside memory in a reservoir specifically meant for bhyve guest memory allocations. This would allow bhyve to quickly and confidently allocate memory for instances without any chance of the ARC (or other large consumers) to soak up free pages, forcing potentially lengthy reaping activity. Use of such a reservoir could be explicitly opt-in, where memory was added to the reservoir via a CLI utility, and bhyve instances were configured to allocate from it when starting. Alternatively, on systems which did not wish to dedicate memory to bhyve instances (particularly when they were not running), bhyve could make "transient" allocations which function similar to things today (where memory is scrounged from the freelist, and reaping from the ARC and kmem).

Related issues

Related to illumos gate - Bug #13896: bhyve VM interfaces should be better fitClosedPatrick Mooney

Actions #1

Updated by Electric Monk over 2 years ago

  • Gerrit CR set to 1522
Actions #2

Updated by Patrick Mooney over 2 years ago

  • Related to Bug #13896: bhyve VM interfaces should be better fit added
Actions #3

Updated by Michael Zeller over 2 years ago

You can find a test platform for SmartOS here (manta currently has an issue so it's stashed on my webserver):

If you wish to see the changes or build them yourself you can find a branch with the commits here:

Actions #4

Updated by Michael Zeller over 2 years ago

The latest bits are now at

The branch has also been updated.

Actions #5

Updated by Patrick Mooney over 2 years ago

Testing for this was a combination of isolated experimentation and end-to-end usage:
- Adding/removing chunks of memory from the reservoir with rsrvrctl
- Starting/stopping VMs which allocate from the reservoir
- Starting/stopping VMs which perform transient allocations (the default behavior for non-reservoir-aware bhyve)
- Bringing the host into situations of memory pressure (while doing the above)
- After other testing was complete, dumping the machine and checking ::findleaks

These tests were run primarily on AMD hardware, although it was checked on an Intel machine as well.

Actions #6

Updated by Patrick Mooney over 2 years ago

It should be noted that the size limit imposed by the reservoir system (shown in rsrvrctl -q as Size limit KiB:) is not adequate for preventing a machine administrator from attempting to add so much memory to the reservoir such that memory pressure makes the system borderline unresponsive. It has been a challenge to find a static calculation that would cover all possible cases, since kernel memory consumption can vary wildly between different hardware configurations. The existing limit was chosen as an attempt to at least keep us out of needfree without being a burden on machines with large amounts of DRAM.

Actions #7

Updated by Electric Monk over 2 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit b57f5d3e6a2df8d435e606797cf3934811848343

commit  b57f5d3e6a2df8d435e606797cf3934811848343
Author: Patrick Mooney <>
Date:   2021-07-27T19:26:22.000Z

    13833 want bhyve memory reservoir
    13822 bhyve memory should exert memory pressure
    13834 want extensible page_resv
    13821 vmmctl ioctls should have more structure
    Reviewed by: Andy Fiddaman <>
    Reviewed by: Jason King <>
    Reviewed by: Dan Cross <>
    Reviewed by: Hans Rosenfeld <>
    Reviewed by: Mike Zeller <>
    Approved by: Dan McDonald <>


Also available in: Atom PDF