Project

General

Profile

Actions

Bug #13659

open

Linux Guest Kernel panic on virtnet_send_command

Added by Till Wegmüller 10 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
bhyve
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

It may be linux bug but somebody has with knowledge to determine if it is or not. Considering that microk8s is a big corporate income source for Canonical, I would assume they tested it with KVM virtio devices.

Steps to preproduce:
Install ubuntu 20.10 from https://releases.ubuntu.com/20.10/ubuntu-20.10-live-server-amd64.iso
Tip: Set "console=ttyS0" on grub first boot from CDROM
Setup networking as wished.
Install microk8s according to https://microk8s.io/

After that wait for the Kernel to panic.

If wished I can also provide zone configs and disk image (ZFS Vol) which is setup and this bug occurs.

oflk8s1 login: [   76.440189] kernel BUG at drivers/net/virtio_net.c:1665!
[   76.442247] invalid opcode: 0000 [#1] SMP NOPTI
[   76.443931] CPU: 2 PID: 2360 Comm: systemd-udevd Tainted: G        W         5.8.0-45-generic #51-Ubuntu
[   76.447353] Hardware name: OpenIndiana OpenIndiana HVM, BIOS 13.0 11/10/2020
[   76.448656] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net]
[   76.449788] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 2b 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 25 a8 9c d7 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41
[   76.453129] RSP: 0018:ffffa32400723ad8 EFLAGS: 00010246
[   76.453998] RAX: ffffffffc02021b4 RBX: ffff9123f0113900 RCX: ffffffffc0202000
[   76.455176] RDX: ffffffffc02021dc RSI: 0000000000000011 RDI: ffff9123f0291400
[   76.456348] RBP: ffffa32400723b70 R08: 0000000000000000 R09: 0000000000000004
[   76.457514] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa32400723b80
[   76.458677] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9123f0291400
[   76.459843] FS:  00007fb9dc3648c0(0000) GS:ffff9123f3d00000(0000) knlGS:0000000000000000
[   76.461134] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   76.462049] CR2: 00007fb9dc8ef401 CR3: 000000031f50a000 CR4: 00000000000406e0
[   76.463174] Call Trace:
[   76.463590]  ? kernel_init_free_pages+0x4a/0x60
[   76.464329]  ? vmap_pte_range+0x110/0x1a0
[   76.464976]  ? _cond_resched+0x1a/0x50
[   76.465570]  virtnet_set_guest_offloads+0x55/0x80 [virtio_net]
[   76.466491]  virtnet_set_features+0x53/0x70 [virtio_net]
[   76.467337]  __netdev_update_features+0xf0/0x510
[   76.468079]  netdev_sync_lower_features+0x7e/0x1c0
[   76.468837]  __netdev_update_features+0x139/0x510
[   76.469584]  ethtool_set_features+0xc5/0x120
[   76.470273]  dev_ethtool+0xce9/0x13e0
[   76.470875]  ? inet_ioctl+0x94/0x1e0
[   76.471472]  ? netdev_run_todo+0x5f/0x210
[   76.472123]  ? netdev_name_node_lookup_rcu+0x6c/0x80
[   76.472936]  ? _cond_resched+0x1a/0x50
[   76.473627]  dev_ioctl+0x2d8/0x360
[   76.474229]  sock_do_ioctl+0x9f/0x140
[   76.474845]  sock_ioctl+0x249/0x3b0
[   76.475425]  ? __secure_computing+0x42/0xe0
[   76.476106]  ksys_ioctl+0x8e/0xc0
[   76.476643]  __x64_sys_ioctl+0x1a/0x20
[   76.477246]  do_syscall_64+0x49/0xc0
[   76.477824]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   76.478629] RIP: 0033:0x7fb9dc81631b
[   76.479206] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1d 3b 0d 00 f7 d8 64 89 01 48
[   76.482139] RSP: 002b:00007fff347c3518 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[   76.483324] RAX: ffffffffffffffda RBX: 0000563b0ad4d638 RCX: 00007fb9dc81631b
[   76.484526] RDX: 00007fff347c35a0 RSI: 0000000000008946 RDI: 0000000000000006
[   76.485806] RBP: 00007fff347c3600 R08: 0000563b0c2a4530 R09: 0000563b0c2a4530
[   76.487097] R10: 00007fb9dc8eac50 R11: 0000000000000246 R12: 0000000000000000
[   76.488404] R13: 00000000ffffffff R14: 00007fff347c35a0 R15: 0000563b0c2a4530
[   76.489693] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_net ip_set_hash_ip ip_set veth nfnetlink xfrm_user xfrm_algo overlay xt_nat xt_tcpudp xt_MASQUERADE xt_addrtype xt_mark xt_conntrack iptable_nat nf_nat iptable_mangle aufs xt_comment iptable_filter bpfilter br_netfilter bridge nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua amd_energy input_leds serio_raw mac_hid efi_pstore sch_fq_codel stp llc ip_vs_wrr ip_vs_sh ip_vs_rr ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 drm ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper virtio_blk psmouse virtio_net net_failover failover
[   76.503144] ---[ end trace 76e5320c7d4ca450 ]---
[   76.507298] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net]
[   76.508578] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 2b 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 25 a8 9c d7 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41
[   76.512081] RSP: 0018:ffffa32400723ad8 EFLAGS: 00010246
[   76.512987] RAX: ffffffffc02021b4 RBX: ffff9123f0113900 RCX: ffffffffc0202000
[   76.514235] RDX: ffffffffc02021dc RSI: 0000000000000011 RDI: ffff9123f0291400
[   76.515556] RBP: ffffa32400723b70 R08: 0000000000000000 R09: 0000000000000004
[   76.516796] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa32400723b80
[   76.517972] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9123f0291400
[   76.519140] FS:  00007fb9dc3648c0(0000) GS:ffff9123f3d00000(0000) knlGS:0000000000000000
[   76.520513] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   76.521527] CR2: 00007fb9dc8ef401 CR3: 000000031f50a000 CR4: 00000000000406e0
[   76.522786] Kernel panic - not syncing: Fatal exception
[   76.524778] Kernel Offset: 0x16000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   76.530024] Rebooting in 10 seconds..

Actions #1

Updated by Igor Kozhukhov 10 months ago

what is distribution where you can see panic with virtio/vioif on kvm vm?

Actions #2

Updated by Till Wegmüller 10 months ago

sorry for the confusion

I am running bhyve on OpenIndiana illumos-gate stock.

The comment about KVM is that I don't believe It occurs there, considering it is right after start of microk8s and canonical deploys their Vm's usually with virtio.

Actions #3

Updated by Till Wegmüller 9 months ago

Small Update.

It is still happening on Kernel 5.11 but not with docker, only with Kubernetes.

Actions #4

Updated by Jim Crumpler 8 months ago

I've been running into the same problem for the last 6 months or so.. I assume it's more of a Linux problem tripped up by something different in BHYVE? Enabling microk8s/kubernetes on a BHYVE VM running a recent Linux kernel causes Linux to crash in virtio_net.. continuously crash loop a few minutes after rebooting. Appears to be in 5.x kernels in virtnet_set_guest_offloads.. Not sure if this is related - https://lore.kernel.org/lkml/20200105081111-mutt-send-email-mst@kernel.org/

As a quick workaround - can anyone find where microk8s is trying to set the offload details? What virtio is trying to do is a bit beyond my understanding.

smartos-live release-20210408
illumos-joyent release-20210408

Debian GNU/Linux 10 (buster) - kernel 4.19.0-8-amd64 - OK
Debian GNU/Linux 11 (bullseye) - kernel 5.10.0-6-amd64 - Crash
Ubuntu 18.04.5 LTS - kernel 4.15.0-143-generic - OK
Ubuntu 20.04.2 LTS - kernel 5.4.0-73-generic - Crash

microk8s v1.21.0 (and various a few versions before that)

Debian crash with kernel 5.10

[   84.716356] kernel BUG at drivers/net/virtio_net.c:1665!
[   84.719203] invalid opcode: 0000 [#1] SMP PTI
[   84.721391] CPU: 3 PID: 2939 Comm: systemd-udevd Tainted: G        W         5.10.0-6-amd64 #1 Debian 5.10.28-1
[   84.726457] Hardware name: Joyent SmartDC HVM, BIOS 13.0 11/10/2020
[   84.730150] RIP: 0010:virtnet_send_command+0x140/0x150 [virtio_net]
[   84.733487] Code: 8b 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4c 24 68 65 48 2b 0c 25 28 00 00 00 75 0f 48 83 c4 70 5b 5d 41 5c 41 5d 41 5e c3 <0f> 0b e8 59 b7 6e c8 66 0f 1f 84 00
 00 00 00 00 0f 1f 44 00 00 55
[   84.744994] RSP: 0018:ffffa5b00088fb18 EFLAGS: 00010246
[   84.747728] RAX: ffffffffc01ce1b4 RBX: ffff8999876db940 RCX: ffffffffc01ce000
[   84.751386] RDX: ffffffffc01ce1dc RSI: 0000000000000011 RDI: ffff899c3a89dc00
[   84.755156] RBP: ffffa5b00088fbb8 R08: ffff899987f84000 R09: ffff8999876db0a8
[   84.759000] R10: 0000000000000000 R11: 0000000000000000 R12: ffff899c3a89dc00
[   84.762543] R13: 0000000000000005 R14: 0000000000000000 R15: 00000000ffffffff
[   84.766197] FS:  00007f6ea43478c0(0000) GS:ffff899cafd80000(0000) knlGS:0000000000000000
[   84.770347] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   84.773411] CR2: 00007ffd7ccdcff0 CR3: 0000000106fb8006 CR4: 00000000000706e0
[   84.777141] Call Trace:
[   84.778485]  ? get_page_from_freelist+0x10cd/0x1320
[   84.781210]  virtnet_set_guest_offloads+0x50/0x80 [virtio_net]
[   84.784095]  virtnet_set_features+0x4e/0x70 [virtio_net]
[   84.786607]  __netdev_update_features+0x260/0xa30
[   84.788817]  ? map_kernel_range_noflush+0x278/0x360
[   84.791095]  __netdev_update_features+0x301/0xa30
[   84.793404]  ? security_capable+0x36/0x50
[   84.795341]  dev_ethtool+0x1ce9/0x2860
[   84.797120]  ? __switch_to_asm+0x42/0x70
[   84.799011]  ? __switch_to+0x114/0x450
[   84.800840]  dev_ioctl+0x156/0x480
[   84.802478]  sock_do_ioctl+0x9b/0x130
[   84.804310]  sock_ioctl+0x240/0x3b0
[   84.806054]  __x64_sys_ioctl+0x83/0xb0
[   84.807864]  do_syscall_64+0x33/0x80
[   84.809636]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   84.812044] RIP: 0033:0x7f6ea47fccc7
[   84.813846] Code: 00 00 00 48 8b 05 c9 91 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d
 99 91 0c 00 f7 d8 64 89 01 48
[   84.822797] RSP: 002b:00007ffd7ccdebc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[   84.826379] RAX: ffffffffffffffda RBX: 0000560a94cc6918 RCX: 00007f6ea47fccc7
[   84.829800] RDX: 00007ffd7ccdec50 RSI: 0000o_net net_failover virtio_blk failover xhci_pci xhci_hcd crct10dif_pclmul crct10dif_common usbcore crc32_pclmul psmouse crc32c_intel virti
o_pci virtio_ring usb_common virtio
[   84.882392] ---[ end trace 54d4cba83f996844 ]---
[   84.884656] RIP: 0010:virtnet_send_command+0x140/0x150 [virtio_net]
[   84.889286] Code: 8b 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4c 24 68 65 48 2b 0c 25 28 00 00 00 75 0f 48 83 c4 70 5b 5d 41 5c 41 5d 41 5e c3 <0f> 0b e8 59 b7 6e c8 66 0f 1f 84 00
 00 00 00 00 0f 1f 44 00 00 55
[   84.899928] RSP: 0018:ffffa5b00088fb18 EFLAGS: 00010246
[   84.902714] RAX: ffffffffc01ce1b4 RBX: ffff8999876db940 RCX: ffffffffc01ce000
[   84.907249] RDX: ffffffffc01ce1dc RSI: 0000000000000011 RDI: ffff899c3a89dc00
[   84.911461] RBP: ffffa5b00088fbb8 R08: ffff899987f84000 R09: ffff8999876db0a8
[   84.915722] R10: 0000000000000000 R11: 0000000000000000 R12: ffff899c3a89dc00
[   84.919412] R13: 0000000000000005 R14: 0000000000000000 R15: 00000000ffffffff
[   84.924237] FS:  00007f6ea43478c0(0000) GS:ffff899cafd80000(0000) knlGS:0000000000000000
[   84.929274] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   84.932193] CR2: 00007ffd7ccdcff0 CR3: 0000000106fb8006 CR4: 00000000000706e0
[   84.935693] Kernel panic - not syncing: Fatal exception
[   84.940683] Kernel Offset: 0x7000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   84.947310] Rebooting in 10 seconds..

Ubuntu crash with kernel 5.4

[  130.539090] kernel BUG at drivers/net/virtio_net.c:1644!
[  130.543103] invalid opcode: 0000 [#1] SMP PTI
[  130.546325] CPU: 3 PID: 3290 Comm: systemd-udevd Tainted: G        W         5.4.0-73-generic #82-Ubuntu
[  130.552753] Hardware name: Joyent SmartDC HVM, BIOS 13.0 11/10/2020
[  130.558046] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net]
[  130.562034] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 e5 24 9b dc 0f 1f 44 00 00
 0f 1f 44 00 00 55 48 89 e5 41
[  130.572806] RSP: 0018:ffffbf83005f3a28 EFLAGS: 00010246
[  130.576308] RAX: ffffffffc00f01b4 RBX: ffff9e291d8cf8c0 RCX: ffffffffc00f0000
[  130.580947] RDX: ffffffffc00f01dc RSI: 0000000000000011 RDI: ffff9e292d7f4800
[  130.585649] RBP: ffffbf83005f3ac0 R08: ffff9e291fbdc000 R09: ffffbf83005f3ad0
[  130.590518] R10: 0000000000000000 R11: 0000000000000000 R12: ffffbf83005f3ad0
[  130.595234] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9e292d7f4800
[  130.599993] FS:  00007fcd0d91a880(0000) GS:ffff9e292fb80000(0000) knlGS:0000000000000000
[  130.605760] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  130.609872] CR2: 00007fcd0df1f598 CR3: 0000000426880006 CR4: 00000000000606e0
[  130.614782] Call Trace:
[  130.617011]  ? __kmalloc_node_track_caller+0x20a/0x320
[  130.620776]  ? sg_init_table+0x15/0x40
[  130.623583]  virtnet_set_guest_offloads+0x57/0x80 [virtio_net]
[  130.627714]  virtnet_set_features+0x53/0x70 [virtio_net]
[  130.631525]  __netdev_update_features+0x279/0x9f0
[  130.634991]  ? vunmap_page_range+0x2c8/0x390
[  130.638135]  __netdev_update_features+0x308/0x9f0
[  130.641585]  dev_ethtool+0x15b9/0x2950
[  130.644518]  ? sock_sendmsg+0x65/0x70
[  130.647892]  ? _cond_resched+0x19/0x30
[  130.650799]  dev_ioctl+0x13a/0x470
[  130.653506]  sock_do_ioctl+0xa8/0x140
[  130.656591]  sock_ioctl+0x24f/0x3c0
[  130.659395]  do_vfs_ioctl+0x407/0x670
[  130.662230]  ? __secure_computing+0x42/0xe0
[  130.665459]  ksys_ioctl+0x67/0x90
[  130.668137]  __x64_sys_ioctl+0x1a/0x20
[  130.670994]  do_syscall_64+0x57/0x190
[  130.673828]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  130.677354] RIP: 0033:0x7fcd0de9850b
[  130.680186] Code: 0f 1e fa 48 8b 05 85 39 0d 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d
 55 39 0d 00 f7 d8 64 89 01 48
[  130.691773] RSP: 002b:00007ffc616ca0c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  130.696917] RAX: ffffffffffffffda RBX: 0000556555f32238 RCX: 00007fcd0de9850b
[  130.701882] RDX: 00007ffc616ca150 RSI: 0000000000008946 RDI: 0000000000000006
[  130.706727] RBP: 00007ffc616ca1b0 R08: 0000556557ae1050 R09: 0000556557abe840
[  130.711490] R10: 0000556557ab5010 R11: 0000000000000246 R12: 0000000000000000
[  130.716361] R13: 00000000ffffffff R14: 0000556557ae1050 R15: 00007ffc616ca150
[  130.722395] Modules linked in: vxlan xt_multiport ip6_udp_tunnel iptable_raw udp_tunnel ip_set_hash_ip ip_set_hash_net ip_set xfrm_user xfrm_algo veth overlay xt_addrtype xt_nat xt_
tcpudp xt_MASQUERADE xt_mark xt_conntrack ip6table_filter iptable_nat ip6table_mangle iptable_mangle ip6table_nat nf_nat ip6_tables aufs br_netfilter bridge nft_counter nft_compat nf_t
ables nfnetlink iptable_filter xt_comment bpfilter nls_iso8859_1 intel_rapl_msr intel_rapl_common joydev sb_edac rapl input_leds serio_raw mac_hid sch_fq_codel stp llc ip_vs_wrr ip_vs_
sh ip_vs_rr ip_vs nf_conntrack drm nf_defrag_ipv6 nf_defrag_ipv4 sunrpc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_p
q libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper psmouse virtio_net net_failover virtio_blk failove
r hid_generic usbhid hid
[  130.771719] ---[ end trace ba0770a6cca4ae0d ]---
[  130.775291] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net]
[  130.779667] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 33 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 e5 24 9b dc 0f 1f 44 00 00
 0f 1f 44 00 00 55 48 89 e5 41
[  130.786694] RSP: 0018:ffffbf83005f3a28 EFLAGS: 00010246
[  130.788575] RAX: ffffffffc00f01b4 RBX: ffff9e291d8cf8c0 RCX: ffffffffc00f0000
[  130.791754] RDX: ffffffffc00f01dc RSI: 0000000000000011 RDI: ffff9e292d7f4800
[  130.794703] RBP: ffffbf83005f3ac0 R08: ffff9e291fbdc000 R09: ffffbf83005f3ad0
[  130.798221] R10: 0000000000000000 R11: 0000000000000000 R12: ffffbf83005f3ad0
[  130.800561] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9e292d7f4800
[  130.802874] FS:  00007fcd0d91a880(0000) GS:ffff9e292fb80000(0000) knlGS:0000000000000000
[  130.805715] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  130.807648] CR2: 00007fcd0df1f598 CR3: 0000000426880006 CR4: 00000000000606e0
[  130.810151] Kernel panic - not syncing: Fatal exception
[  130.812286] Kernel Offset: 0x1ba00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  130.815452] Rebooting in 10 seconds..

Actions #5

Updated by Till Wegmüller 8 months ago

Jim Crumpler have you tested the SSL connection with Debian 10? I was not able to establish an SSL connection to the Internet from Inside Kubernetes on Debian 10.

Actions #6

Updated by Marcel Telka 6 months ago

  • Subject changed from Linux Guest Kernet panic on virtnet_send_command to Linux Guest Kernel panic on virtnet_send_command
Actions

Also available in: Atom PDF