Note that I realised this was the case because snoop was failing on the NIC with EIO:

# snoop -r -d usbecm0
snoop: cannot open "usbecm0": I/O error

I dug into this with DTrace, and found roughly where the problem arose:

# dtrace -F -n 'fbt:usbecm::entry { printf(" "); self->x = 1; }'  -n 'fbt:usbecm::return { self->x = 0; trace((int)arg1); }' -n 'fbt::usb*:entry /self->x/ { printf(" "); } fbt::usb*:return /self->x/ { trace((int)arg1); }'
...

    CPU FUNCTION
      0  -> usbecm_m_start
      0   | usbecm_m_start:entry
      0    -> usb_serialize_access
      0    <- usb_serialize_access                        1
      0    -> usbecm_open_pipes
      0     | usbecm_open_pipes:entry
...
      0          <- usba_sync_set_alt_if                -14
      0        <- usba_pipe_setup_func_call             -14
      0      <- usb_set_alt_if                          -14
      0      -> usb_dprintf2
      0        -> usba_vlog
      0          -> usb_vprintf
      0          <- usb_vprintf                           0
      0        <- usba_vlog                               0
      0      <- usb_dprintf2                              0
      0    <- usbecm_open_pipes                         -14
      0  <- usbecm_m_start                                5

In this context, -14 is USB_INVALID_PERM ("dip does not own the interface to be set." according to the manual). Indeed this appeared to be the case, because we were requesting an interface that was indeed part of the IA, but from the wrong dip: the interface itself, where we had bound in error, rather than the IA one node up.

Testing Notes¶

I built the change and installed it. I looked at prtconf output in four combinations: before and after the change, with the device removed and inserted. This highlights the nodes that are specific to the device in both cases and allows us to compare easily:

Before Change¶

$ diff -u before_prtconf_{removed,inserted}.txt
--- before_prtconf_removed.txt    2021-06-26 23:01:31.443136450 -0700
+++ before_prtconf_inserted.txt    2021-06-26 23:02:05.103053833 -0700
@@ -10,6 +10,13 @@
         pci8086,2064 (pciex8086,1911) [Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model]
         pci8086,2064 (pciex8086,a12f) [Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller], instance #0 (driver name: xhci)
             device, instance #1 (driver name: usb_mid)
+            communications, instance #4 (driver name: usb_mid)
+                interface-association, instance #0 (driver name: usb_ia)
+                    ethernet, instance #0 (driver name: usbecm)
+                    data
+                storage, instance #5 (driver name: scsa2usb)
+                    disk, instance #8 (driver name: sd)
+                    disk, instance #9 (driver name: sd)
         pci8086,2064 (pciex8086,a131) [Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem]
         pci8086,2064 (pciex8086,a13a) [Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1]
         pci8086,a110 (pciex8086,a110) [Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1], instance #0 (driver name: pcieb)

After Change¶

$ diff -u after_prtconf_{removed,inserted}.txt
--- after_prtconf_removed.txt    2021-06-26 23:10:06.222021457 -0700
+++ after_prtconf_inserted.txt    2021-06-26 23:10:50.645438977 -0700
@@ -10,6 +10,11 @@
         pci8086,2064 (pciex8086,1911) [Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model]
         pci8086,2064 (pciex8086,a12f) [Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller], instance #0 (driver name: xhci)
             device, instance #1 (driver name: usb_mid)
+            communications, instance #4 (driver name: usb_mid)
+                interface-association, instance #1 (driver name: usbecm)
+                storage, instance #5 (driver name: scsa2usb)
+                    disk, instance #8 (driver name: sd)
+                    disk, instance #9 (driver name: sd)
         pci8086,2064 (pciex8086,a131) [Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem]
         pci8086,2064 (pciex8086,a13a) [Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1]
         pci8086,a110 (pciex8086,a110) [Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1], instance #0 (driver name: pcieb)

Note that the alias is also present, and snoop works correctly:

$ grep usbecm /etc/driver_aliases
usbecm "usb,class2.6.0" 
usbecm "usb430,a4a2" 
usbecm "usbif,class2.6" 
usbecm "usbia,class2.6" 

$ pfexec snoop -r -d usbecm1
Using device usbecm1 (promiscuous mode)
 UNSPECIFIED -> ff02::16     ICMPv6 Group membership report - MLDv2
 UNSPECIFIED -> ff02::16     ICMPv6 Group membership report - MLDv2
 UNSPECIFIED -> ff02::1:ffb7:ef08 ICMPv6 Neighbor solicitation
fe80::364b:50ff:feb7:ef08 -> ff02::16     ICMPv6 Group membership report - MLDv2
fe80::364b:50ff:feb7:ef08 -> ff02::2      ICMPv6 Router solicitation
fe80::364b:50ff:feb7:ef08 -> ff02::16     ICMPv6 Group membership report - MLDv2
...

Comparing Before & After¶

$ diff -u {before,after}_prtconf_inserted.txt
--- before_prtconf_inserted.txt    2021-06-26 23:02:05.103053833 -0700
+++ after_prtconf_inserted.txt    2021-06-26 23:10:50.645438977 -0700
@@ -11,9 +11,7 @@
         pci8086,2064 (pciex8086,a12f) [Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller], instance #0 (driver name: xhci)
             device, instance #1 (driver name: usb_mid)
             communications, instance #4 (driver name: usb_mid)
-                interface-association, instance #0 (driver name: usb_ia)
-                    ethernet, instance #0 (driver name: usbecm)
-                    data
+                interface-association, instance #1 (driver name: usbecm)
                 storage, instance #5 (driver name: scsa2usb)
                     disk, instance #8 (driver name: sd)
                     disk, instance #9 (driver name: sd)