Actions
Bug #14536
closed
bhyve: viona uses wrong address for queue notify ioport
Start date:
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:
Description
After #14372, viona is failing to properly register the ioport hook for virtqueue notification. This is a result of the upstream change to defer BAR allocation until all of the requirements have been collected:
https://github.com/freebsd/freebsd-src/commit/01f9362ef4eb14b041ccdf935fccf0f794074258
Using truss to observe ioctl(VNA_IOC_SET_NOTIFY_IOP) for a VM with two viona NICs, shows this:
/1: ioctl(4, (('V'<<16)|('C'<<8)|35), 0x00000010) = 0
/1: ioctl(6, (('V'<<16)|('C'<<8)|35), 0x00000010) Err#17 EEXIST
The bar address being used here is just 0, as it has not yet been allocated.
whereas with an older system:
/1: ioctl(6, (('V'<<16)|('C'<<8)|35), 0x00002010) = 0
/1: ioctl(7, (('V'<<16)|('C'<<8)|35), 0x00002030) = 0
Related issues
Updated by Andy Fiddaman over 1 year ago
- Related to Feature #14372: bhyve upstream sync 2022 January added
Updated by Andy Fiddaman over 1 year ago
- Subject changed from bhyve virtio-viona fails to install the virtqueue notification ioport hook to bhyve: viona uses wrong address for queue notify ioport
- Category set to bhyve
Updated by Andy Fiddaman over 1 year ago
Following the change, things look much more reasonable:
/1: ioctl(4, (('V'<<16)|('C'<<8)|35), 0x00002010) = 0
/1: ioctl(6, (('V'<<16)|('C'<<8)|35), 0x00002030) = 0
Updated by Michael Zeller over 1 year ago
Dan cherry-picked this change into SmartOS.
Prior to the change I was unable to start a VM with more than one viona based vnic:
{ "time": "2022-02-24T16:23:41.153600000Z", "v": 0, "hostname": "chico", "name": "zoneadmd","pid": 10161, "level": 30, "msg":"device emulation initialization error: File exists\n", "stream":"stderr" }
{ "time": "2022-02-24T16:23:41.153625000Z", "v": 0, "hostname": "chico", "name": "zoneadmd","pid": 10161, "level": 30, "msg":"could not install ioport hook at 10\n", "stream":"stdout" }
{ "time": "2022-02-24T16:23:42.440504000Z", "v": 0, "hostname": "chico", "name": "zoneadmd","pid": 10161, "level": 30, "stream":"logfile", "msg":"close" }
After this change:
net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 86.109.7.90 netmask 255.255.255.248 broadcast 86.109.7.95
inet6 fe80::92b8:d0ff:feb0:6c4c prefixlen 64 scopeid 0x20<link>
ether 90:b8:d0:b0:6c:4c txqueuelen 1000 (Ethernet)
RX packets 7824 bytes 11338445 (11.3 MB)
RX errors 0 dropped 99 overruns 0 frame 0
TX packets 1750 bytes 137347 (137.3 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
net1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 8192
inet 192.168.128.22 netmask 255.255.252.0 broadcast 192.168.131.255
inet6 fe80::92b8:d0ff:fe90:6761 prefixlen 64 scopeid 0x20<link>
ether 90:b8:d0:90:67:61 txqueuelen 1000 (Ethernet)
RX packets 1 bytes 60 (60.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 8 bytes 656 (656.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ubuntu@47eca903-6a0f-43e1-b23c-d43f391ca72b:~$ ping 192.168.128.1 PING 192.168.128.1 (192.168.128.1) 56(84) bytes of data. 64 bytes from 192.168.128.1: icmp_seq=1 ttl=255 time=0.099 ms 64 bytes from 192.168.128.1: icmp_seq=2 ttl=255 time=0.096 ms ^C --- 192.168.128.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 0.096/0.097/0.099/0.010 ms ubuntu@47eca903-6a0f-43e1-b23c-d43f391ca72b:~$ ping 86.109.7.89 PING 86.109.7.89 (86.109.7.89) 56(84) bytes of data. 64 bytes from 86.109.7.89: icmp_seq=1 ttl=64 time=0.125 ms 64 bytes from 86.109.7.89: icmp_seq=2 ttl=64 time=0.141 ms ^C --- 86.109.7.89 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1018ms rtt min/avg/max/mdev = 0.125/0.133/0.141/0.008 ms
Finally I verified that the viona nics were using ioports rather than falling back to poll(2):
[root@chico (ducksoup) ~]# dtrace -n 'fbt::viona*intr*:entry' dtrace: description 'fbt::viona*intr*:entry' matched 3 probes CPU ID FUNCTION:NAME 15 5903 viona_intr_ring:entry 14 5903 viona_intr_ring:entry 15 5903 viona_intr_ring:entry 15 5903 viona_intr_ring:entry
[root@chico (ducksoup) ~]# dtrace -n 'fbt::viona*chpoll:entry' dtrace: description 'fbt::viona*chpoll:entry' matched 1 probe ^C
Updated by Andy Fiddaman over 1 year ago
I also tested this by watching for calls to viona's viona_notify_iop() function in the host kernel. Before the patch, no calls were observed and afterwards they were:
# dtrace -n 'viona_notify_iop:entry' dtrace: description 'viona_notify_iop:entry' matched 1 probe CPU ID FUNCTION:NAME 0 89371 viona_notify_iop:entry 3 89371 viona_notify_iop:entry 4 89371 viona_notify_iop:entry 1 89371 viona_notify_iop:entry 3 89371 viona_notify_iop:entry 7 89371 viona_notify_iop:entry 5 89371 viona_notify_iop:entry
Updated by Electric Monk over 1 year ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
git commit f24fee035ef9b37d5a6868aed10261da6316a6b2
commit f24fee035ef9b37d5a6868aed10261da6316a6b2
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date: 2022-02-24T19:09:44.000Z
14536 bhyve: viona uses wrong address for queue notify ioport
Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
Reviewed by: Patrick Mooney <pmooney@pfmooney.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Actions