Project

General

Profile

Bug #8579

ixgbe infinite loops in rar and vmdq clearing

Added by Robert Mustacchi over 3 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Category:
driver - device drivers
Start date:
2017-08-14
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

We've seen a case where we have an ixgbe fail in the following way:

ffffd003d7acf030 ixgbe_clear_vmdq_generic+0xd()
ffffd003d7acf070 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf0e0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf120 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf190 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf1d0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf240 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf280 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf2f0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf330 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf3a0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf3e0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf450 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf490 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf500 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf540 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf5b0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf5f0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf660 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf6a0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf710 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf750 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf7c0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf800 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf870 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf8b0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf920 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acf960 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acf9d0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfa10 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acfa80 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfac0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acfb30 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfb70 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acfbe0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfc20 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acfc90 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfcd0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acfd40 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfd80 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acfdf0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfe30 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acfea0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acfee0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7acff50 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7acff90 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0000 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0040 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad00b0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad00f0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0160 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad01a0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0210 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0250 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad02c0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0300 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0370 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad03b0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0420 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0460 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad04d0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0510 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0580 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad05c0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0630 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0670 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad06e0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0720 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0790 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad07d0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0840 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0880 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad08f0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0930 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad09a0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad09e0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0a50 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0a90 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0b00 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0b40 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0bb0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0bf0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0c60 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0ca0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0d10 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0d50 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0dc0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0e00 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0e70 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0eb0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0f20 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad0f60 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad0fd0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1010 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1080 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad10c0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1130 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1170 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad11e0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1220 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1290 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad12d0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1340 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1380 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad13f0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1430 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad14a0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad14e0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1550 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1590 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1600 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1640 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad16b0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad16f0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1760 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad17a0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1810 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1850 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad18c0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1900 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1970 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad19b0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1a20 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1a60 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1ad0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1b10 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1b80 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1bc0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1c30 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1c70 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1ce0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1d20 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1d90 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1dd0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1e40 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1e80 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1ef0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1f30 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad1fa0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad1fe0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2050 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2090 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2100 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2140 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad21b0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad21f0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2260 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad22a0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2310 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2350 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad23c0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2400 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2470 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad24b0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2520 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2560 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad25d0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2610 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2680 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad26c0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2730 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2770 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad27e0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2820 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2890 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad28d0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2940 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2980 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad29f0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2a30 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2aa0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2ae0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2b50 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2b90 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2c00 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2c40 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2cb0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2cf0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2d60 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2da0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2e10 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2e50 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2ec0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2f00 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad2f70 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad2fb0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3020 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3060 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad30d0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3110 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3180 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad31c0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3230 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3270 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad32e0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3320 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3390 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad33d0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3440 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3480 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad34f0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3530 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad35a0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad35e0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3650 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3690 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3700 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad3740 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad37b0 ixgbe_clear_vmdq_generic+0x139(ffffd0c572ed3018, 1, ffffffff)
ffffd003d7ad37f0 ixgbe_clear_rar_generic+0xa7(ffffd0c572ed3018, 1)
ffffd003d7ad3810 ixgbe_clear_rar+0x23(ffffd0c572ed3018, 1)
ffffd003d7ad3860 ixgbe_remmac+0xb0(ffffd0c572e926d8, ffffd0c572ec7f50)
ffffd003d7ad3880 mac_group_remmac+0x1d(ffffd0c572f0c090, ffffd0c572ec7f50)
ffffd003d7ad38c0 mac_remove_macaddr+0xa5(ffffd0c572ec7f40)
ffffd003d7ad3960 mac_datapath_teardown+0xbe(ffffd0c572e1b8f0, ffffd0c572e5e5e0, 1)
ffffd003d7ad39c0 mac_client_datapath_teardown+0x66(ffffd0c572e1b8f0, ffffd0c573799458, 0)
ffffd003d7ad3a10 mac_unicast_remove+0x27a(ffffd0c572e1b8f0, ffffd0c573799458)
ffffd003d7ad3a40 dls_mac_active_clear+0x38(ffffd0c572df06e8)
ffffd003d7ad3a70 dls_active_clear+0x46(ffffd0c58727f728, 1)
ffffd003d7ad3ab0 dls_close+0x96(ffffd0c58727f728)
ffffd003d7ad3af0 dld_str_detach+0x5f(ffffd0c58727f728)
ffffd003d7ad3b30 dld_str_close+0x6a(ffffd0c79765dad0)
ffffd003d7ad3b60 dld_close+0x20(ffffd0c79765dad0)
ffffd003d7ad3be0 qdetach+0x9b(ffffd0c79765dad0, 1, 3, ffffd0c584cf00c8, 0)
ffffd003d7ad3c80 strclose+0x2c1(ffffd0c57428ee80, 3, ffffd0c584cf00c8)
ffffd003d7ad3cd0 device_close+0xb9(ffffd0c573fd9b40, 3, ffffd0c584cf00c8)
ffffd003d7ad3d60 spec_close+0x17b(ffffd0c573fd9b40, 3, 1, 0, ffffd0c584cf00c8, 0)
ffffd003d7ad3de0 fop_close+0x61(ffffd0c573fd9b40, 3, 1, 0, ffffd0c584cf00c8, 0)
ffffd003d7ad3e20 closef+0x5e(ffffd0c570bcc7e8)
ffffd003d7ad3e90 closeandsetf+0x398(7, 0)
ffffd003d7ad3eb0 close+0x13(7)
ffffd003d7ad3f10 _sys_sysenter_post_swapgs+0x153()
> *panic_thread::print kthread_t t_procp
t_procp = 0xffffd0c583637028
> *panic_thread::print kthread_t t_procp->p_user.u_comm
t_procp->p_user.u_comm = [ "lldpd" ]

So, we had an instance of lldpd being restarted or closed and something seems to have gone wildly wrong in the driver.

I had an idea of what we could potentially do here. We could implement the IXGBE_REMOVED macro by doing a register read of the STATUS register which has several bits which are both reserved and hardcoded to zero. We could also do a register access check potentially and if FM complains also use this to fail the operation. Though the latter we'll have to be careful with due to how things work in the surrounding.

I confirmed in the datasheet for the 82598, 82599 (X520), X540 and X550 that we'll always have the reserved bits available, so this scheme should work.

To test this I booted a system that had ixgbe and caused these entry points to be fired by running the mc_start/mc_stop(9E) entry points. The easiest way to do this was to restart the lldp service without plumbing up the interfaces. Using DTrace I verified that all the calls to ixgbe_removed() always returned false, the same as we'd expect. Unfortunately we don't have a great way of actually testing the NIC disappearing in this way at this time.

Because of the contexts that this is running in, I did not opt to end up doing a check with FM about the register accesses. Because this is a bit buried, I didn't want to end up stealing the data that others might end up relying on later on. While this is a bit unfortunate, it doesn't seem the worst given the realities of this failure mode.

To test that we properly bump counters and complain when running, I instrumented with kmdb and changed the register return value to be PCI_EINVAL32.

#1

Updated by Robert Mustacchi over 3 years ago

  • Subject changed from ixgbe inifinte loops in rar and vmdq clearing to ixgbe infinite loops in rar and vmdq clearing
#2

Updated by Electric Monk over 3 years ago

  • Status changed from New to Closed

git commit 74b989c3047109458504db6bc5b1caec29c43ff3

commit  74b989c3047109458504db6bc5b1caec29c43ff3
Author: Robert Mustacchi <rm@joyent.com>
Date:   2017-09-21T20:48:43.000Z

    8579 ixgbe infinite loops in rar and vmdq clearing
    Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
    Reviewed by: Michael Speer <michael.speer@pluribusnetworks.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom PDF