Feature #12529
closedwant exclusive hma registration
100%
Description
Upstreaming for SmartOS change OS-7600:
Andy Fiddaman from OmniOSce is working on wiring the in-kernel bits of VirtualBox into HMA. While it may be difficult to make it play nice with other hypervisors simultaneously, giving it an exclusive HMA registration would likely be fine.
Testing notes¶
Some testing notes for this change on OmniOS are below.
Having gone through this, I think that it is also necessary to have bhyve perform hma_registration/deregistration upon starting the first and stopping the last VM, otherwise the vmm module needs to be not installed, or disabled in etc/system or boot parameters.
On an OmniOS bloody host which has Bhyve, KVM and Virtualbox installed, all drivers are loaded:
theeo% modinfo | egrep 'vmm|kvm|vboxdrv' 187 fffffffff7f7c000 56f48 275 1 vboxdrv (VirtualBox HostDrv 5.2.26r12828) 267 fffffffff8388000 23b320 273 1 vmm (vmm) 279 fffffffff7ce1000 39970 288 1 kvm (kvm driver v0.1)
With no VMs running, only bhyve has an active registration:
theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' const char *hr_name = 0xfffffffff83b47ad "bhyve"
Starting a VirtualBox VM creates an additional registration:
theeo# VBoxHeadless -s test & theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' const char *hr_name = 0xfffffffff83b47ad "bhyve" const char *hr_name = 0xfffffffff7fb3abe "VirtualBox HostDrv"
Using a version of VirtualBox which uses the new hma_register_exclusive():
theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' const char *hr_name = 0xfffffffff7f987ad "bhyve" theeo# VBoxHeadless -s test Oracle VM VirtualBox Headless Interface 5.2.26 (C) 2008-2019 Oracle Corporation All rights reserved. 18/03/2019 13:59:18 Listening for VNC connections on TCP port 5900 18/03/2019 13:59:18 Listening for VNC connections on TCP6 port 5900 VRDE server is listening on port 5900. Error: failed to start machine. Error message: VT-x is being used by another hypervisor (VERR_VMX_IN_VMX_ROOT_MODE). VirtualBox can't operate in VMX root mode. Please close all other virtualization programs. (VERR_VMX_IN_VMX_ROOT_MODE) 18/03/2019 13:59:18 listenerRun: error in select: Bad file number theeo# echo exclude: vmm > /etc/system.d/novmm theeo# init 6 theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' theeo# VBoxHeadless -s test & theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' const char *hr_name = 0xfffffffff7fb2abe "VirtualBox HostDrv" theeo# mdb -kwe 'sysparam_hd/z0' theeo# mdb -kwe 'sysparam_tl/z0' theeo# modload /usr/kernel/drv/amd64/vmm theeo# modload /usr/kernel/drv/amd64/viona INFO:root:/usr/sbin/bhyve -U 4d17cdc5-e409-4cb4-a70a-8ed866c19551 -A -H -B 1,product=OmniOS HVM -c 2 -m 1G -l bootrom,/usr/share/bhyve/firmware/BHYVE_CSM.fd -s 0,hostbridge,model=i440fx -s 1,lpc -l com1,/dev/zconsole -s 4:0,virtio-blk,/dev/zvol/rdsk/data/hdd-bloody -s 6:0,virtio-net-viona,bloody0 bloody INFO:root:Starting bhyve INFO:root:Bhyve exited 4 ERROR:root:Error b'vm_create: No such file or directory\n' DEBUG:root:Output b''
...this is because the sdev plugin never registered due to HMA registration failure in vmm_attach()
INFO:root:/usr/bin/qemu-system-x86_64 -name bloody -uuid 4d17cdc5-e409-4cb4-a70a-8ed866c19551 -enable-kvm -no-hpet -m 1G -smp 2 -cpu qemu64 -rtc base=utc,driftfix=slew -pidfile /tmp/vm.pid -monitor unix:/tmp/vm.monitor,server,nowait,nodelay -vga std -chardev pipe,id=console0,path=/dev/zconsole -serial chardev:console0 -boot order=dc -drive file=/dev/zvol/rdsk/data/hdd-bloody,if=virtio,media=disk,index=0,cache=none,serial=0,boot=on -device virtio-net-pci,mac=02:08:20:ab:f4:27,tx=timer,x-txtimer=200000,x-txburst=128,vlan=0 -net vnic,vlan=0,name=net0,ifname=bloody0 -vnc none INFO:root:Starting kvm INFO:root:KVM exited 1 ERROR:root:Error b'open /dev/kvm: No such device or address\nfailed to initialize KVM: Not owner\n' DEBUG:root:Output b'' theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' const char *hr_name = 0xfffffffff7fb2abe "VirtualBox HostDrv"
.... stop Virtualbox VM
theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' INFO:root:/usr/bin/qemu-system-x86_64 -name bloody -uuid 4d17cdc5-e409-4cb4-a70a-8ed866c19551 -enable-kvm -no-hpet -m 1G -smp 2 -cpu qemu64 -rtc base=utc,driftfix=slew -pidfile /tmp/vm.pid -monitor unix:/tmp/vm.monitor,server,nowait,nodelay -vga std -chardev pipe,id=console0,path=/dev/zconsole -serial chardev:console0 -boot order=dc -drive file=/dev/zvol/rdsk/data/hdd-bloody,if=virtio,media=disk,index=0,cache=none,serial=0,boot=on -device virtio-net-pci,mac=02:08:20:ab:f4:27,tx=timer,x-txtimer=200000,x-txburst=128,vlan=0 -net vnic,vlan=0,name=net0,ifname=bloody0 -vnc none INFO:root:Starting kvm theeo# mdb -ke 'hma_registrations::walk list | ::print -t hma_reg_t hr_name' const char *hr_name = 0xfffffffff7d0d361 "SmartOS KVM"
Related issues
Updated by Patrick Mooney about 2 years ago
- Related to Feature #12528: expand hypervisor management API added
Updated by Electric Monk about 2 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit 454f0c49f9d5b08ab88fe3db5788d9e5e6a7cf0f
commit 454f0c49f9d5b08ab88fe3db5788d9e5e6a7cf0f Author: Andy Fiddaman <omnios@citrus-it.co.uk> Date: 2020-04-28T16:44:52.000Z 12529 want exclusive hma registration Reviewed by: Patrick Mooney <patrick.mooney@joyent.com> Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com> Reviewed by: Toomas Soome <tsoome@me.com> Approved by: Dan McDonald <danmcd@joyent.com>