Project

General

Profile

Actions

Bug #14837

closed

panic with max_rx_set() on VNIC with secondary macs

Added by Andy Fiddaman 2 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
networking
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

While booting DEBUG bits to test something else on a machine which has a few bhyve VMs set to auto boot I encountered this panic:

> ::status
debugging crash dump vmcore.1 (64-bit) from reaper
operating system: 5.11 omnios-rmpcie-659fec57e1 (i86pc)
build version: gfx-drm - heads/master-0-g77f745e
illumos-kvm - heads/master-0-g93cd164
heads/rmpcie-0-g659fec57e1

image uuid: 171e618f-c587-cee0-e0dd-a6c2dfeb3212
panic message: assertion failed: MAC_PERIM_HELD((mac_handle_t)mip), file: ../../common/io/mac/mac_client.c, line: 2078
dump content: kernel pages and pages from PID -1
> $C
fffffbe3670ca8a0 vpanic()
fffffbe3670ca8f0 0xfffffffffbdd1b35()
fffffbe3670ca960 mac_resource_ctl_set+0x198(fffffdf8a0313b90, fffffdf8a02d1124)
fffffbe3670ca990 mac_secondary_dup+0x5e(fffffdf8a02ceb90, fffffdf8a0313b90)
fffffbe3670ca9c0 vnic_mac_secondary_update+0x32(fffffdf893c56588)
fffffbe3670caa00 mac_vnic_secondary_update+0x38(fffffdf8a02e5fc0)
fffffbe3670caa60 mac_rx_set+0x83(fffffdf8a02ceb90, fffffffff7fd4590, fffffdfbd8b35250)
fffffbe3670caaa0 viona_rx_set+0x2d(fffffdfbd8b35240)
fffffbe3670cac20 viona_ioc_create+0x28f(fffffdfbd8ce1348, fffffaffffdf8d20, 202403, fffffdf8dff30850)
fffffbe3670cacb0 viona_ioctl+0x119(11500000001, 564301, fffffaffffdf8d20, 202403, fffffdf8dff30850, fffffbe3670cae08)
fffffbe3670cacf0 cdev_ioctl+0x3f(11500000001, 564301, fffffaffffdf8d20, 202403, fffffdf8dff30850, fffffbe3670cae08)
fffffbe3670cad40 spec_ioctl+0x55(fffffdfbf24e7200, 564301, fffffaffffdf8d20, 202403, fffffdf8dff30850, fffffbe3670cae08, 0)
fffffbe3670cadd0 fop_ioctl+0x40(fffffdfbf24e7200, 564301, fffffaffffdf8d20, 202403, fffffdf8dff30850, fffffbe3670cae08, 0)
fffffbe3670caef0 ioctl+0x144(9, 564301, fffffaffffdf8d20)
fffffbe3670caf00 sys_syscall+0x2ae()

The VNIC which is being bound to viona has some secondary MAC addresses set on it, like

dladm set-linkprop -p secondary-macs=a:0:0:0:0:1 <vnic>


Related issues

Related to illumos gate - Feature #5013: add support for multiple mac addresses per clientClosedJerry Jelinek2014-07-17

Actions
Actions #1

Updated by Andy Fiddaman 2 months ago

  • Related to Feature #5013: add support for multiple mac addresses per client added
Actions #2

Updated by Andy Fiddaman 2 months ago

  • Assignee set to Andy Fiddaman
Actions #3

Updated by Andy Fiddaman 2 months ago

It looks like this is just an early perim exit in mac_rx_set(), I moved the exit down below the secondary MAC address setup and the system no-longer panics.

Testing:

Using viona from a simple test program since it's an easy way to drive mac_rx_set (and where the original panic was triggered)

VNIC with no secondary MACs

bloody# dladm reset-linkprop -p secondary-macs mac0
bloody# dtrace -Fn 'mac_resource_ctl_set:' &
[1] 102394

bloody# ./test mac0
Using VNIC 'mac0'
Created VIONA instance, sleeping for 10 seconds...
..^Z
zsh: suspended  ./test mac0
bloody# dladm set-linkprop -p secondary-macs=a:0:0:0:0:2 mac0
bloody# CPU FUNCTION
  1  -> mac_resource_ctl_set
  1  <- mac_resource_ctl_set
bloody# dladm reset-linkprop -p secondary-macs mac0
bloody#
bloody# fg
[2]  - continued  ./test mac0
........Closing VIONA

VNIC with one secondary MAC

bloody# dladm set-linkprop -p secondary-macs=a:0:0:0:0:1 mac0
bloody#  12  -> mac_resource_ctl_set
 12  <- mac_resource_ctl_set

bloody# ./test mac0
Using VNIC 'mac0'
Created VIONA instance, sleeping for 10 seconds...
.  7  -> mac_resource_ctl_set
  7  <- mac_resource_ctl_set
  7  -> mac_resource_ctl_set
  7  <- mac_resource_ctl_set
.........Closing VIONA
bloody#   7  -> mac_resource_ctl_set
  7  <- mac_resource_ctl_set
  7  -> mac_resource_ctl_set
  7  <- mac_resource_ctl_set

VNIC with two secondary MACs

bloody# dladm set-linkprop -p secondary-macs=a:0:0:0:0:1,a:0:0:0:0:2 mac0
bloody#  11  -> mac_resource_ctl_set
 11  <- mac_resource_ctl_set
 11  -> mac_resource_ctl_set
 11  <- mac_resource_ctl_set

bloody# ./test mac0
Using VNIC 'mac0'
Created VIONA instance, sleeping for 10 seconds...
.  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
.........Closing VIONA
bloody#   0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set

VNIC with two secondary MACs, removing one while viona is open

bloody# dladm set-linkprop -p secondary-macs=a:0:0:0:0:1,a:0:0:0:0:2 mac0
bloody#  13  -> mac_resource_ctl_set
 13  <- mac_resource_ctl_set
 14  -> mac_resource_ctl_set
 14  <- mac_resource_ctl_set

bloody# ./test mac0
Using VNIC 'mac0'
Created VIONA instance, sleeping for 10 seconds...
.  8  -> mac_resource_ctl_set
  8  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
  0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
.^Z
zsh: suspended  ./test mac0
bloody# dladm set-linkprop -p secondary-macs=a:0:0:0:0:1 mac0
bloody#  10  -> mac_resource_ctl_set
 10  <- mac_resource_ctl_set

bloody# fg
[2]  - continued  ./test mac0
........Closing VIONA
bloody#   0  -> mac_resource_ctl_set
  0  <- mac_resource_ctl_set
 10  -> mac_resource_ctl_set
 10  <- mac_resource_ctl_set

Checking that reset-linkprop does remove the additional MACs

Since I saw less dtrace output than I expected, I also verified that reset-linkprop is doing the right thing here

bloody# dladm set-linkprop -p secondary-macs=a:0:0:0:0:1,a:0:0:0:0:2 mac0

> *viona_state::walk softstate_all | ::grep .>0 | ::print viona_soft_state_t ss_link | ::print viona_link_t l_mch | ::print mac_client_impl_t mci_mip | ::print mac_impl_t mi_addresses | ::list mac_address_t ma_next | ::print mac_address_t ma_addr
ma_addr = [ 0xa, 0, 0, 0, 0, 0x2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ma_addr = [ 0xa, 0, 0, 0, 0, 0x1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ma_addr = [ 0x2, 0x8, 0x20, 0xb0, 0x6f, 0x10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ma_addr = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]

bloody# dladm reset-linkprop -p secondary-macs mac0

> *viona_state::walk softstate_all | ::grep .>0 | ::print viona_soft_state_t ss_link | ::print viona_link_t l_mch | ::print mac_client_impl_t mci_mip | ::print mac_impl_t mi_addresses | ::list mac_address_t ma_next | ::print mac_address_t ma_addr
ma_addr = [ 0x2, 0x8, 0x20, 0xb0, 0x6f, 0x10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
ma_addr = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ]
Actions #4

Updated by Electric Monk 2 months ago

  • Gerrit CR set to 2259
Actions #5

Updated by Electric Monk about 1 month ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 119d61ccb993f400b23aa47a130a03fd12b5aabb

commit  119d61ccb993f400b23aa47a130a03fd12b5aabb
Author: Andy Fiddaman <illumos@fiddaman.net>
Date:   2022-08-20T20:07:00.000Z

    14837 panic with max_rx_set() on VNIC with secondary macs
    Reviewed by: Dan McDonald <danmcd@mnx.io>
    Reviewed by: Ryan Zezeski <ryan@oxide.computer>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Actions

Also available in: Atom PDF