Project

General

Profile

Bug #13659

Linux Guest Kernet panic on virtnet_send_command

Added by Till Wegmüller about 1 month ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
bhyve
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

It may be linux bug but somebody has with knowledge to determine if it is or not. Considering that microk8s is a big corporate income source for Canonical, I would assume they tested it with KVM virtio devices.

Steps to preproduce:
Install ubuntu 20.10 from https://releases.ubuntu.com/20.10/ubuntu-20.10-live-server-amd64.iso
Tip: Set "console=ttyS0" on grub first boot from CDROM
Setup networking as wished.
Install microk8s according to https://microk8s.io/

After that wait for the Kernel to panic.

If wished I can also provide zone configs and disk image (ZFS Vol) which is setup and this bug occurs.

oflk8s1 login: [   76.440189] kernel BUG at drivers/net/virtio_net.c:1665!
[   76.442247] invalid opcode: 0000 [#1] SMP NOPTI
[   76.443931] CPU: 2 PID: 2360 Comm: systemd-udevd Tainted: G        W         5.8.0-45-generic #51-Ubuntu
[   76.447353] Hardware name: OpenIndiana OpenIndiana HVM, BIOS 13.0 11/10/2020
[   76.448656] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net]
[   76.449788] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 2b 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 25 a8 9c d7 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41
[   76.453129] RSP: 0018:ffffa32400723ad8 EFLAGS: 00010246
[   76.453998] RAX: ffffffffc02021b4 RBX: ffff9123f0113900 RCX: ffffffffc0202000
[   76.455176] RDX: ffffffffc02021dc RSI: 0000000000000011 RDI: ffff9123f0291400
[   76.456348] RBP: ffffa32400723b70 R08: 0000000000000000 R09: 0000000000000004
[   76.457514] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa32400723b80
[   76.458677] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9123f0291400
[   76.459843] FS:  00007fb9dc3648c0(0000) GS:ffff9123f3d00000(0000) knlGS:0000000000000000
[   76.461134] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   76.462049] CR2: 00007fb9dc8ef401 CR3: 000000031f50a000 CR4: 00000000000406e0
[   76.463174] Call Trace:
[   76.463590]  ? kernel_init_free_pages+0x4a/0x60
[   76.464329]  ? vmap_pte_range+0x110/0x1a0
[   76.464976]  ? _cond_resched+0x1a/0x50
[   76.465570]  virtnet_set_guest_offloads+0x55/0x80 [virtio_net]
[   76.466491]  virtnet_set_features+0x53/0x70 [virtio_net]
[   76.467337]  __netdev_update_features+0xf0/0x510
[   76.468079]  netdev_sync_lower_features+0x7e/0x1c0
[   76.468837]  __netdev_update_features+0x139/0x510
[   76.469584]  ethtool_set_features+0xc5/0x120
[   76.470273]  dev_ethtool+0xce9/0x13e0
[   76.470875]  ? inet_ioctl+0x94/0x1e0
[   76.471472]  ? netdev_run_todo+0x5f/0x210
[   76.472123]  ? netdev_name_node_lookup_rcu+0x6c/0x80
[   76.472936]  ? _cond_resched+0x1a/0x50
[   76.473627]  dev_ioctl+0x2d8/0x360
[   76.474229]  sock_do_ioctl+0x9f/0x140
[   76.474845]  sock_ioctl+0x249/0x3b0
[   76.475425]  ? __secure_computing+0x42/0xe0
[   76.476106]  ksys_ioctl+0x8e/0xc0
[   76.476643]  __x64_sys_ioctl+0x1a/0x20
[   76.477246]  do_syscall_64+0x49/0xc0
[   76.477824]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   76.478629] RIP: 0033:0x7fb9dc81631b
[   76.479206] Code: 89 d8 49 8d 3c 1c 48 f7 d8 49 39 c4 72 b5 e8 1c ff ff ff 85 c0 78 ba 4c 89 e0 5b 5d 41 5c c3 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 1d 3b 0d 00 f7 d8 64 89 01 48
[   76.482139] RSP: 002b:00007fff347c3518 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[   76.483324] RAX: ffffffffffffffda RBX: 0000563b0ad4d638 RCX: 00007fb9dc81631b
[   76.484526] RDX: 00007fff347c35a0 RSI: 0000000000008946 RDI: 0000000000000006
[   76.485806] RBP: 00007fff347c3600 R08: 0000563b0c2a4530 R09: 0000563b0c2a4530
[   76.487097] R10: 00007fb9dc8eac50 R11: 0000000000000246 R12: 0000000000000000
[   76.488404] R13: 00000000ffffffff R14: 00007fff347c35a0 R15: 0000563b0c2a4530
[   76.489693] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_net ip_set_hash_ip ip_set veth nfnetlink xfrm_user xfrm_algo overlay xt_nat xt_tcpudp xt_MASQUERADE xt_addrtype xt_mark xt_conntrack iptable_nat nf_nat iptable_mangle aufs xt_comment iptable_filter bpfilter br_netfilter bridge nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua amd_energy input_leds serio_raw mac_hid efi_pstore sch_fq_codel stp llc ip_vs_wrr ip_vs_sh ip_vs_rr ip_vs nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 drm ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper virtio_blk psmouse virtio_net net_failover failover
[   76.503144] ---[ end trace 76e5320c7d4ca450 ]---
[   76.507298] RIP: 0010:virtnet_send_command+0x144/0x150 [virtio_net]
[   76.508578] Code: 83 d8 00 00 00 80 78 02 00 0f 94 c0 48 8b 4d d0 65 48 2b 0c 25 28 00 00 00 75 11 48 83 c4 70 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <0f> 0b e8 25 a8 9c d7 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41
[   76.512081] RSP: 0018:ffffa32400723ad8 EFLAGS: 00010246
[   76.512987] RAX: ffffffffc02021b4 RBX: ffff9123f0113900 RCX: ffffffffc0202000
[   76.514235] RDX: ffffffffc02021dc RSI: 0000000000000011 RDI: ffff9123f0291400
[   76.515556] RBP: ffffa32400723b70 R08: 0000000000000000 R09: 0000000000000004
[   76.516796] R10: 0000000000000000 R11: 0000000000000000 R12: ffffa32400723b80
[   76.517972] R13: 0000000000000000 R14: 0000000000000005 R15: ffff9123f0291400
[   76.519140] FS:  00007fb9dc3648c0(0000) GS:ffff9123f3d00000(0000) knlGS:0000000000000000
[   76.520513] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   76.521527] CR2: 00007fb9dc8ef401 CR3: 000000031f50a000 CR4: 00000000000406e0
[   76.522786] Kernel panic - not syncing: Fatal exception
[   76.524778] Kernel Offset: 0x16000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   76.530024] Rebooting in 10 seconds..

#1

Updated by Igor Kozhukhov about 1 month ago

what is distribution where you can see panic with virtio/vioif on kvm vm?

#2

Updated by Till Wegmüller about 1 month ago

sorry for the confusion

I am running bhyve on OpenIndiana illumos-gate stock.

The comment about KVM is that I don't believe It occurs there, considering it is right after start of microk8s and canonical deploys their Vm's usually with virtio.

Also available in: Atom PDF