illumos gate

Upstreaming for SmartOS change OS-7600:

Andy Fiddaman from OmniOSce is working on wiring the in-kernel bits of VirtualBox into HMA. While it may be difficult to make it play nice with other hypervisors simultaneously, giving it an exclusive HMA registration would likely be fine.

Testing notes¶

Some testing notes for this change on OmniOS are below.

Having gone through this, I think that it is also necessary to have bhyve perform hma_registration/deregistration upon starting the first and stopping the last VM, otherwise the vmm module needs to be not installed, or disabled in etc/system or boot parameters.

On an OmniOS bloody host which has Bhyve, KVM and Virtualbox installed, all drivers are loaded:

theeo% modinfo | egrep 'vmm|kvm|vboxdrv'
187 fffffffff7f7c000  56f48 275   1  vboxdrv (VirtualBox HostDrv 5.2.26r12828)
267 fffffffff8388000 23b320 273   1  vmm (vmm)
279 fffffffff7ce1000  39970 288   1  kvm (kvm driver v0.1)

With no VMs running, only bhyve has an active registration:

theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'
const char *hr_name = 0xfffffffff83b47ad "bhyve" 

Starting a VirtualBox VM creates an additional registration:

theeo# VBoxHeadless -s test &
theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'
const char *hr_name = 0xfffffffff83b47ad "bhyve" 
const char *hr_name = 0xfffffffff7fb3abe "VirtualBox HostDrv" 

Using a version of VirtualBox which uses the new hma_register_exclusive():

theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'
const char *hr_name = 0xfffffffff7f987ad "bhyve" 

theeo# VBoxHeadless -s test
Oracle VM VirtualBox Headless Interface 5.2.26
(C) 2008-2019 Oracle Corporation
All rights reserved.

18/03/2019 13:59:18 Listening for VNC connections on TCP port 5900
18/03/2019 13:59:18 Listening for VNC connections on TCP6 port 5900
VRDE server is listening on port 5900.
Error: failed to start machine. Error message: VT-x is being used by another hypervisor (VERR_VMX_IN_VMX_ROOT_MODE).
VirtualBox can't operate in VMX root mode. Please close all other virtualization programs. (VERR_VMX_IN_VMX_ROOT_MODE)
18/03/2019 13:59:18 listenerRun: error in select: Bad file number

theeo# echo exclude: vmm > /etc/system.d/novmm
theeo# init 6

theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'
theeo# VBoxHeadless -s test &
theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'
const char *hr_name = 0xfffffffff7fb2abe "VirtualBox HostDrv" 

theeo# mdb -kwe 'sysparam_hd/z0'
theeo# mdb -kwe 'sysparam_tl/z0'
theeo# modload /usr/kernel/drv/amd64/vmm
theeo# modload /usr/kernel/drv/amd64/viona

INFO:root:/usr/sbin/bhyve -U 4d17cdc5-e409-4cb4-a70a-8ed866c19551 -A -H -B 1,product=OmniOS HVM -c 2 -m 1G -l bootrom,/usr/share/bhyve/firmware/BHYVE_CSM.fd -s 0,hostbridge,model=i440fx -s 1,lpc -l com1,/dev/zconsole -s 4:0,virtio-blk,/dev/zvol/rdsk/data/hdd-bloody -s 6:0,virtio-net-viona,bloody0 bloody
INFO:root:Starting bhyve
INFO:root:Bhyve exited 4
ERROR:root:Error b'vm_create: No such file or directory\n'
DEBUG:root:Output b''

...this is because the sdev plugin never registered due to HMA registration failure in vmm_attach()

INFO:root:/usr/bin/qemu-system-x86_64 -name bloody -uuid 4d17cdc5-e409-4cb4-a70a-8ed866c19551 -enable-kvm -no-hpet -m 1G -smp 2 -cpu qemu64 -rtc base=utc,driftfix=slew -pidfile /tmp/vm.pid -monitor unix:/tmp/vm.monitor,server,nowait,nodelay -vga std -chardev pipe,id=console0,path=/dev/zconsole -serial chardev:console0 -boot order=dc -drive file=/dev/zvol/rdsk/data/hdd-bloody,if=virtio,media=disk,index=0,cache=none,serial=0,boot=on -device virtio-net-pci,mac=02:08:20:ab:f4:27,tx=timer,x-txtimer=200000,x-txburst=128,vlan=0 -net vnic,vlan=0,name=net0,ifname=bloody0 -vnc none
INFO:root:Starting kvm
INFO:root:KVM exited 1
ERROR:root:Error b'open /dev/kvm: No such device or address\nfailed to initialize KVM: Not owner\n'
DEBUG:root:Output b''

theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'
const char *hr_name = 0xfffffffff7fb2abe "VirtualBox HostDrv" 

.... stop Virtualbox VM

theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'

INFO:root:/usr/bin/qemu-system-x86_64 -name bloody -uuid 4d17cdc5-e409-4cb4-a70a-8ed866c19551 -enable-kvm -no-hpet -m 1G -smp 2 -cpu qemu64 -rtc base=utc,driftfix=slew -pidfile /tmp/vm.pid -monitor unix:/tmp/vm.monitor,server,nowait,nodelay -vga std -chardev pipe,id=console0,path=/dev/zconsole -serial chardev:console0 -boot order=dc -drive file=/dev/zvol/rdsk/data/hdd-bloody,if=virtio,media=disk,index=0,cache=none,serial=0,boot=on -device virtio-net-pci,mac=02:08:20:ab:f4:27,tx=timer,x-txtimer=200000,x-txburst=128,vlan=0 -net vnic,vlan=0,name=net0,ifname=bloody0 -vnc none
INFO:root:Starting kvm

theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name'
const char *hr_name = 0xfffffffff7d0d361 "SmartOS KVM"