Actions
Bug #13659
open
Linux Guest Kernel panic on virtnet_send_command
Status:
New
Priority:
Normal
Assignee:
-
Category:
bhyve
Start date:
Due date:
% Done:
0%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:
Description
It may be linux bug but somebody has with knowledge to determine if it is or not. Considering that microk8s is a big corporate income source for Canonical, I would assume they tested it with KVM virtio devices.
Steps to preproduce:
Install ubuntu 20.10 from https://releases.ubuntu.com/20.10/ubuntu-20.10-live-server-amd64.iso
Tip: Set "console=ttyS0" on grub first boot from CDROM
Setup networking as wished.
Install microk8s according to https://microk8s.io/
After that wait for the Kernel to panic.
If wished I can also provide zone configs and disk image (ZFS Vol) which is setup and this bug occurs.
oflk8s1 login: [ 76.440189] kernel BUG at drivers/net/virtio_net.c:1665! [ 76.442247] invalid opcode: 0000 [#1] SMP NOPTI [ 76.443931] CPU: 2 PID: 2360 Comm: systemd-udevd Tainted: G W 5.8.0-45-generic #51-Ubuntu [ 76.447353] Hardware name: OpenIndiana OpenIndiana HVM, BIOS 13.0 11/10/2020 [ 76.448656] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net] [ 76.449788] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 2b 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 25 a8 9c d7 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41 [ 76.453129] RSP: 0018:ffffa32400723ad8 EFLAGS: 00010246 [ 76.453998] RAX: ffffffffc02021b4 RBX: ffff9123f0113900 RCX: ffffffffc0202000 [ 76.455176] RDX: ffffffffc02021dc RSI: 0000000000000011 RDI: ffff9123f0291400 [ 76.456348] RBP: ffffa32400723b70 R08: 0000000000000000 R09: 0000000000000004 [ 76.457514] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa32400723b80 [ 76.458677] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9123f0291400 [ 76.459843] FS: 00007fb9dc3648c0(0000) GS:ffff9123f3d00000(0000) knlGS:0000000000000000 [ 76.461134] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 76.462049] CR2: 00007fb9dc8ef401 CR3: 000000031f50a000 CR4: 00000000000406e0 [ 76.463174] Call Trace: [ 76.463590] ? kernel_init_free_pages+0x4a/0x60 [ 76.464329] ? vmap_pte_range+0x110/0x1a0 [ 76.464976] ? _cond_resched+0x1a/0x50 [ 76.465570] virtnet_set_guest_offloads+0x55/0x80 [virtio_net] [ 76.466491] virtnet_set_features+0x53/0x70 [virtio_net] [ 76.467337] __netdev_update_features+0xf0/0x510 [ 76.468079] netdev_sync_lower_features+0x7e/0x1c0 [ 76.468837] __netdev_update_features+0x139/0x510 [ 76.469584] ethtool_set_features+0xc5/0x120 [ 76.470273] dev_ethtool+0xce9/0x13e0 [ 76.470875] ? inet_ioctl+0x94/0x1e0 [ 76.471472] ? netdev_run_todo+0x5f/0x210 [ 76.472123] ? netdev_name_node_lookup_rcu+0x6c/0x80 [ 76.472936] ? _cond_resched+0x1a/0x50 [ 76.473627] dev_ioctl+0x2d8/0x360 [ 76.474229] sock_do_ioctl+0x9f/0x140 [ 76.474845] sock_ioctl+0x249/0x3b0 [ 76.475425] ? __secure_computing+0x42/0xe0 [ 76.476106] ksys_ioctl+0x8e/0xc0 [ 76.476643] __x64_sys_ioctl+0x1a/0x20 [ 76.477246] do_syscall_64+0x49/0xc0 [ 76.477824] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 76.478629] RIP: 0033:0x7fb9dc81631b [ 76.479206] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1d 3b 0d 00 f7 d8 64 89 01 48 [ 76.482139] RSP: 002b:00007fff347c3518 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 76.483324] RAX: ffffffffffffffda RBX: 0000563b0ad4d638 RCX: 00007fb9dc81631b [ 76.484526] RDX: 00007fff347c35a0 RSI: 0000000000008946 RDI: 0000000000000006 [ 76.485806] RBP: 00007fff347c3600 R08: 0000563b0c2a4530 R09: 0000563b0c2a4530 [ 76.487097] R10: 00007fb9dc8eac50 R11: 0000000000000246 R12: 0000000000000000 [ 76.488404] R13: 00000000ffffffff R14: 00007fff347c35a0 R15: 0000563b0c2a4530 [ 76.489693] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_net ip_set_hash_ip ip_set veth nfnetlink xfrm_user xfrm_algo overlay xt_nat xt_tcpudp xt_MASQUERADE xt_addrtype xt_mark xt_conntrack iptable_nat nf_nat iptable_mangle aufs xt_comment iptable_filter bpfilter br_netfilter bridge nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua amd_energy input_leds serio_raw mac_hid efi_pstore sch_fq_codel stp llc ip_vs_wrr ip_vs_sh ip_vs_rr ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 drm ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper virtio_blk psmouse virtio_net net_failover failover [ 76.503144] ---[ end trace 76e5320c7d4ca450 ]--- [ 76.507298] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net] [ 76.508578] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 2b 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 25 a8 9c d7 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41 [ 76.512081] RSP: 0018:ffffa32400723ad8 EFLAGS: 00010246 [ 76.512987] RAX: ffffffffc02021b4 RBX: ffff9123f0113900 RCX: ffffffffc0202000 [ 76.514235] RDX: ffffffffc02021dc RSI: 0000000000000011 RDI: ffff9123f0291400 [ 76.515556] RBP: ffffa32400723b70 R08: 0000000000000000 R09: 0000000000000004 [ 76.516796] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa32400723b80 [ 76.517972] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9123f0291400 [ 76.519140] FS: 00007fb9dc3648c0(0000) GS:ffff9123f3d00000(0000) knlGS:0000000000000000 [ 76.520513] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 76.521527] CR2: 00007fb9dc8ef401 CR3: 000000031f50a000 CR4: 00000000000406e0 [ 76.522786] Kernel panic - not syncing: Fatal exception [ 76.524778] Kernel Offset: 0x16000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 76.530024] Rebooting in 10 seconds..
Updated by Igor Kozhukhov about 2 years ago
what is distribution where you can see panic with virtio/vioif on kvm vm?
Updated by Till Wegmüller about 2 years ago
sorry for the confusion
I am running bhyve on OpenIndiana illumos-gate stock.
The comment about KVM is that I don't believe It occurs there, considering it is right after start of microk8s and canonical deploys their Vm's usually with virtio.
Updated by Till Wegmüller about 2 years ago
Small Update.
It is still happening on Kernel 5.11 but not with docker, only with Kubernetes.
Updated by Jim Crumpler about 2 years ago
I've been running into the same problem for the last 6 months or so.. I assume it's more of a Linux problem tripped up by something different in BHYVE? Enabling microk8s/kubernetes on a BHYVE VM running a recent Linux kernel causes Linux to crash in virtio_net.. continuously crash loop a few minutes after rebooting. Appears to be in 5.x kernels in virtnet_set_guest_offloads.. Not sure if this is related - https://lore.kernel.org/lkml/20200105081111-mutt-send-email-mst@kernel.org/
As a quick workaround - can anyone find where microk8s is trying to set the offload details? What virtio is trying to do is a bit beyond my understanding.
smartos-live release-20210408
illumos-joyent release-20210408
Debian GNU/Linux 10 (buster) - kernel 4.19.0-8-amd64 - OK
Debian GNU/Linux 11 (bullseye) - kernel 5.10.0-6-amd64 - Crash
Ubuntu 18.04.5 LTS - kernel 4.15.0-143-generic - OK
Ubuntu 20.04.2 LTS - kernel 5.4.0-73-generic - Crash
microk8s v1.21.0 (and various a few versions before that)
Debian crash with kernel 5.10
[ 84.716356] kernel BUG at drivers/net/virtio_net.c:1665! [ 84.719203] invalid opcode: 0000 [#1] SMP PTI [ 84.721391] CPU: 3 PID: 2939 Comm: systemd-udevd Tainted: G W 5.10.0-6-amd64 #1 Debian 5.10.28-1 [ 84.726457] Hardware name: Joyent SmartDC HVM, BIOS 13.0 11/10/2020 [ 84.730150] RIP: 0010:virtnet_send_command+0x140/0x150 [virtio_net] [ 84.733487] Code: 8b 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4c 24 68 65 48 2b 0c 25 28 00 00 00 75 0f 48 83 c4 70 5b 5d 41 5c 41 5d 41 5e c3 <0f> 0b e8 59 b7 6e c8 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [ 84.744994] RSP: 0018:ffffa5b00088fb18 EFLAGS: 00010246 [ 84.747728] RAX: ffffffffc01ce1b4 RBX: ffff8999876db940 RCX: ffffffffc01ce000 [ 84.751386] RDX: ffffffffc01ce1dc RSI: 0000000000000011 RDI: ffff899c3a89dc00 [ 84.755156] RBP: ffffa5b00088fbb8 R08: ffff899987f84000 R09: ffff8999876db0a8 [ 84.759000] R10: 0000000000000000 R11: 0000000000000000 R12: ffff899c3a89dc00 [ 84.762543] R13: 0000000000000005 R14: 0000000000000000 R15: 00000000ffffffff [ 84.766197] FS: 00007f6ea43478c0(0000) GS:ffff899cafd80000(0000) knlGS:0000000000000000 [ 84.770347] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 84.773411] CR2: 00007ffd7ccdcff0 CR3: 0000000106fb8006 CR4: 00000000000706e0 [ 84.777141] Call Trace: [ 84.778485] ? get_page_from_freelist+0x10cd/0x1320 [ 84.781210] virtnet_set_guest_offloads+0x50/0x80 [virtio_net] [ 84.784095] virtnet_set_features+0x4e/0x70 [virtio_net] [ 84.786607] __netdev_update_features+0x260/0xa30 [ 84.788817] ? map_kernel_range_noflush+0x278/0x360 [ 84.791095] __netdev_update_features+0x301/0xa30 [ 84.793404] ? security_capable+0x36/0x50 [ 84.795341] dev_ethtool+0x1ce9/0x2860 [ 84.797120] ? __switch_to_asm+0x42/0x70 [ 84.799011] ? __switch_to+0x114/0x450 [ 84.800840] dev_ioctl+0x156/0x480 [ 84.802478] sock_do_ioctl+0x9b/0x130 [ 84.804310] sock_ioctl+0x240/0x3b0 [ 84.806054] __x64_sys_ioctl+0x83/0xb0 [ 84.807864] do_syscall_64+0x33/0x80 [ 84.809636] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 84.812044] RIP: 0033:0x7f6ea47fccc7 [ 84.813846] Code: 00 00 00 48 8b 05 c9 91 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 99 91 0c 00 f7 d8 64 89 01 48 [ 84.822797] RSP: 002b:00007ffd7ccdebc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 84.826379] RAX: ffffffffffffffda RBX: 0000560a94cc6918 RCX: 00007f6ea47fccc7 [ 84.829800] RDX: 00007ffd7ccdec50 RSI: 0000o_net net_failover virtio_blk failover xhci_pci xhci_hcd crct10dif_pclmul crct10dif_common usbcore crc32_pclmul psmouse crc32c_intel virti o_pci virtio_ring usb_common virtio [ 84.882392] ---[ end trace 54d4cba83f996844 ]--- [ 84.884656] RIP: 0010:virtnet_send_command+0x140/0x150 [virtio_net] [ 84.889286] Code: 8b 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4c 24 68 65 48 2b 0c 25 28 00 00 00 75 0f 48 83 c4 70 5b 5d 41 5c 41 5d 41 5e c3 <0f> 0b e8 59 b7 6e c8 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 [ 84.899928] RSP: 0018:ffffa5b00088fb18 EFLAGS: 00010246 [ 84.902714] RAX: ffffffffc01ce1b4 RBX: ffff8999876db940 RCX: ffffffffc01ce000 [ 84.907249] RDX: ffffffffc01ce1dc RSI: 0000000000000011 RDI: ffff899c3a89dc00 [ 84.911461] RBP: ffffa5b00088fbb8 R08: ffff899987f84000 R09: ffff8999876db0a8 [ 84.915722] R10: 0000000000000000 R11: 0000000000000000 R12: ffff899c3a89dc00 [ 84.919412] R13: 0000000000000005 R14: 0000000000000000 R15: 00000000ffffffff [ 84.924237] FS: 00007f6ea43478c0(0000) GS:ffff899cafd80000(0000) knlGS:0000000000000000 [ 84.929274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 84.932193] CR2: 00007ffd7ccdcff0 CR3: 0000000106fb8006 CR4: 00000000000706e0 [ 84.935693] Kernel panic - not syncing: Fatal exception [ 84.940683] Kernel Offset: 0x7000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 84.947310] Rebooting in 10 seconds..
Ubuntu crash with kernel 5.4
[ 130.539090] kernel BUG at drivers/net/virtio_net.c:1644! [ 130.543103] invalid opcode: 0000 [#1] SMP PTI [ 130.546325] CPU: 3 PID: 3290 Comm: systemd-udevd Tainted: G W 5.4.0-73-generic #82-Ubuntu [ 130.552753] Hardware name: Joyent SmartDC HVM, BIOS 13.0 11/10/2020 [ 130.558046] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net] [ 130.562034] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 e5 24 9b dc 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41 [ 130.572806] RSP: 0018:ffffbf83005f3a28 EFLAGS: 00010246 [ 130.576308] RAX: ffffffffc00f01b4 RBX: ffff9e291d8cf8c0 RCX: ffffffffc00f0000 [ 130.580947] RDX: ffffffffc00f01dc RSI: 0000000000000011 RDI: ffff9e292d7f4800 [ 130.585649] RBP: ffffbf83005f3ac0 R08: ffff9e291fbdc000 R09: ffffbf83005f3ad0 [ 130.590518] R10: 0000000000000000 R11: 0000000000000000 R12: ffffbf83005f3ad0 [ 130.595234] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9e292d7f4800 [ 130.599993] FS: 00007fcd0d91a880(0000) GS:ffff9e292fb80000(0000) knlGS:0000000000000000 [ 130.605760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 130.609872] CR2: 00007fcd0df1f598 CR3: 0000000426880006 CR4: 00000000000606e0 [ 130.614782] Call Trace: [ 130.617011] ? __kmalloc_node_track_caller+0x20a/0x320 [ 130.620776] ? sg_init_table+0x15/0x40 [ 130.623583] virtnet_set_guest_offloads+0x57/0x80 [virtio_net] [ 130.627714] virtnet_set_features+0x53/0x70 [virtio_net] [ 130.631525] __netdev_update_features+0x279/0x9f0 [ 130.634991] ? vunmap_page_range+0x2c8/0x390 [ 130.638135] __netdev_update_features+0x308/0x9f0 [ 130.641585] dev_ethtool+0x15b9/0x2950 [ 130.644518] ? sock_sendmsg+0x65/0x70 [ 130.647892] ? _cond_resched+0x19/0x30 [ 130.650799] dev_ioctl+0x13a/0x470 [ 130.653506] sock_do_ioctl+0xa8/0x140 [ 130.656591] sock_ioctl+0x24f/0x3c0 [ 130.659395] do_vfs_ioctl+0x407/0x670 [ 130.662230] ? __secure_computing+0x42/0xe0 [ 130.665459] ksys_ioctl+0x67/0x90 [ 130.668137] __x64_sys_ioctl+0x1a/0x20 [ 130.670994] do_syscall_64+0x57/0x190 [ 130.673828] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [ 130.677354] RIP: 0033:0x7fcd0de9850b [ 130.680186] Code: 0f 1e fa 48 8b 05 85 39 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 55 39 0d 00 f7 d8 64 89 01 48 [ 130.691773] RSP: 002b:00007ffc616ca0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 130.696917] RAX: ffffffffffffffda RBX: 0000556555f32238 RCX: 00007fcd0de9850b [ 130.701882] RDX: 00007ffc616ca150 RSI: 0000000000008946 RDI: 0000000000000006 [ 130.706727] RBP: 00007ffc616ca1b0 R08: 0000556557ae1050 R09: 0000556557abe840 [ 130.711490] R10: 0000556557ab5010 R11: 0000000000000246 R12: 0000000000000000 [ 130.716361] R13: 00000000ffffffff R14: 0000556557ae1050 R15: 00007ffc616ca150 [ 130.722395] Modules linked in: vxlan xt_multiport ip6_udp_tunnel iptable_raw udp_tunnel ip_set_hash_ip ip_set_hash_net ip_set xfrm_user xfrm_algo veth overlay xt_addrtype xt_nat xt_ tcpudp xt_MASQUERADE xt_mark xt_conntrack ip6table_filter iptable_nat ip6table_mangle iptable_mangle ip6table_nat nf_nat ip6_tables aufs br_netfilter bridge nft_counter nft_compat nf_t ables nfnetlink iptable_filter xt_comment bpfilter nls_iso8859_1 intel_rapl_msr intel_rapl_common joydev sb_edac rapl input_leds serio_raw mac_hid sch_fq_codel stp llc ip_vs_wrr ip_vs_ sh ip_vs_rr ip_vs nf_conntrack drm nf_defrag_ipv6 nf_defrag_ipv4 sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_p q libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper psmouse virtio_net net_failover virtio_blk failove r hid_generic usbhid hid [ 130.771719] ---[ end trace ba0770a6cca4ae0d ]--- [ 130.775291] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net] [ 130.779667] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 e5 24 9b dc 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41 [ 130.786694] RSP: 0018:ffffbf83005f3a28 EFLAGS: 00010246 [ 130.788575] RAX: ffffffffc00f01b4 RBX: ffff9e291d8cf8c0 RCX: ffffffffc00f0000 [ 130.791754] RDX: ffffffffc00f01dc RSI: 0000000000000011 RDI: ffff9e292d7f4800 [ 130.794703] RBP: ffffbf83005f3ac0 R08: ffff9e291fbdc000 R09: ffffbf83005f3ad0 [ 130.798221] R10: 0000000000000000 R11: 0000000000000000 R12: ffffbf83005f3ad0 [ 130.800561] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9e292d7f4800 [ 130.802874] FS: 00007fcd0d91a880(0000) GS:ffff9e292fb80000(0000) knlGS:0000000000000000 [ 130.805715] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 130.807648] CR2: 00007fcd0df1f598 CR3: 0000000426880006 CR4: 00000000000606e0 [ 130.810151] Kernel panic - not syncing: Fatal exception [ 130.812286] Kernel Offset: 0x1ba00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 130.815452] Rebooting in 10 seconds..
Updated by Till Wegmüller about 2 years ago
Jim Crumpler have you tested the SSL connection with Debian 10? I was not able to establish an SSL connection to the Internet from Inside Kubernetes on Debian 10.
Updated by Marcel Telka almost 2 years ago
- Subject changed from Linux Guest Kernet panic on virtnet_send_command to Linux Guest Kernel panic on virtnet_send_command
Actions