Project

General

Profile

Actions

Bug #14536

closed

bhyve: viona uses wrong address for queue notify ioport

Added by Andy Fiddaman 7 months ago. Updated 7 months ago.

Status:
Closed
Priority:
High
Assignee:
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

After #14372, viona is failing to properly register the ioport hook for virtqueue notification. This is a result of the upstream change to defer BAR allocation until all of the requirements have been collected:

https://github.com/freebsd/freebsd-src/commit/01f9362ef4eb14b041ccdf935fccf0f794074258

Using truss to observe ioctl(VNA_IOC_SET_NOTIFY_IOP) for a VM with two viona NICs, shows this:

/1:     ioctl(4, (('V'<<16)|('C'<<8)|35), 0x00000010)   = 0
/1:     ioctl(6, (('V'<<16)|('C'<<8)|35), 0x00000010)   Err#17 EEXIST

The bar address being used here is just 0, as it has not yet been allocated.

whereas with an older system:

/1:     ioctl(6, (('V'<<16)|('C'<<8)|35), 0x00002010)   = 0
/1:     ioctl(7, (('V'<<16)|('C'<<8)|35), 0x00002030)   = 0

Related issues

Related to illumos gate - Feature #14372: bhyve upstream sync 2022 JanuaryClosedAndy Fiddaman

Actions
Actions #1

Updated by Andy Fiddaman 7 months ago

Actions #2

Updated by Electric Monk 7 months ago

  • Gerrit CR set to 2053
Actions #3

Updated by Andy Fiddaman 7 months ago

  • Subject changed from bhyve virtio-viona fails to install the virtqueue notification ioport hook to bhyve: viona uses wrong address for queue notify ioport
  • Category set to bhyve
Actions #4

Updated by Andy Fiddaman 7 months ago

Following the change, things look much more reasonable:

/1:     ioctl(4, (('V'<<16)|('C'<<8)|35), 0x00002010)   = 0
/1:     ioctl(6, (('V'<<16)|('C'<<8)|35), 0x00002030)   = 0
Actions #5

Updated by Michael Zeller 7 months ago

Dan cherry-picked this change into SmartOS.

Prior to the change I was unable to start a VM with more than one viona based vnic:

{ "time": "2022-02-24T16:23:41.153600000Z", "v": 0, "hostname": "chico", "name": "zoneadmd","pid": 10161, "level": 30, "msg":"device emulation initialization error: File exists\n", "stream":"stderr" }
{ "time": "2022-02-24T16:23:41.153625000Z", "v": 0, "hostname": "chico", "name": "zoneadmd","pid": 10161, "level": 30, "msg":"could not install ioport hook at 10\n", "stream":"stdout" }
{ "time": "2022-02-24T16:23:42.440504000Z", "v": 0, "hostname": "chico", "name": "zoneadmd","pid": 10161, "level": 30, "stream":"logfile", "msg":"close" }

After this change:

net0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 86.109.7.90  netmask 255.255.255.248  broadcast 86.109.7.95
        inet6 fe80::92b8:d0ff:feb0:6c4c  prefixlen 64  scopeid 0x20<link>
        ether 90:b8:d0:b0:6c:4c  txqueuelen 1000  (Ethernet)
        RX packets 7824  bytes 11338445 (11.3 MB)
        RX errors 0  dropped 99  overruns 0  frame 0
        TX packets 1750  bytes 137347 (137.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

net1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 8192
        inet 192.168.128.22  netmask 255.255.252.0  broadcast 192.168.131.255
        inet6 fe80::92b8:d0ff:fe90:6761  prefixlen 64  scopeid 0x20<link>
        ether 90:b8:d0:90:67:61  txqueuelen 1000  (Ethernet)
        RX packets 1  bytes 60 (60.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 656 (656.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
ubuntu@47eca903-6a0f-43e1-b23c-d43f391ca72b:~$ ping 192.168.128.1
PING 192.168.128.1 (192.168.128.1) 56(84) bytes of data.
64 bytes from 192.168.128.1: icmp_seq=1 ttl=255 time=0.099 ms
64 bytes from 192.168.128.1: icmp_seq=2 ttl=255 time=0.096 ms
^C
--- 192.168.128.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.096/0.097/0.099/0.010 ms
ubuntu@47eca903-6a0f-43e1-b23c-d43f391ca72b:~$ ping 86.109.7.89
PING 86.109.7.89 (86.109.7.89) 56(84) bytes of data.
64 bytes from 86.109.7.89: icmp_seq=1 ttl=64 time=0.125 ms
64 bytes from 86.109.7.89: icmp_seq=2 ttl=64 time=0.141 ms
^C
--- 86.109.7.89 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.125/0.133/0.141/0.008 ms

Finally I verified that the viona nics were using ioports rather than falling back to poll(2):

[root@chico (ducksoup) ~]# dtrace -n 'fbt::viona*intr*:entry'
dtrace: description 'fbt::viona*intr*:entry' matched 3 probes
CPU     ID                    FUNCTION:NAME
 15   5903            viona_intr_ring:entry
 14   5903            viona_intr_ring:entry
 15   5903            viona_intr_ring:entry
 15   5903            viona_intr_ring:entry

[root@chico (ducksoup) ~]# dtrace -n 'fbt::viona*chpoll:entry'
dtrace: description 'fbt::viona*chpoll:entry' matched 1 probe
^C
Actions #6

Updated by Andy Fiddaman 7 months ago

I also tested this by watching for calls to viona's viona_notify_iop() function in the host kernel. Before the patch, no calls were observed and afterwards they were:

# dtrace -n 'viona_notify_iop:entry'
dtrace: description 'viona_notify_iop:entry' matched 1 probe
CPU     ID                    FUNCTION:NAME
  0  89371           viona_notify_iop:entry
  3  89371           viona_notify_iop:entry
  4  89371           viona_notify_iop:entry
  1  89371           viona_notify_iop:entry
  3  89371           viona_notify_iop:entry
  7  89371           viona_notify_iop:entry
  5  89371           viona_notify_iop:entry
Actions #7

Updated by Electric Monk 7 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit f24fee035ef9b37d5a6868aed10261da6316a6b2

commit  f24fee035ef9b37d5a6868aed10261da6316a6b2
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2022-02-24T19:09:44.000Z

    14536 bhyve: viona uses wrong address for queue notify ioport
    Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
    Reviewed by: Patrick Mooney <pmooney@pfmooney.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF