illumos gate

Here is the one from FreeBSD

BdsDxe: loading Boot0001 "UEFI bhyve-NVMe NVME-4-0" from PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-84-87-20-FC-9C-58)
BdsDxe: starting Boot0001 "UEFI bhyve-NVMe NVME-4-0" from PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-84-87-20-FC-9C-58)
Consoles: EFI console
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk0p1:
FreeBSD/amd64 EFI loader, Revision 1.1

   Command line arguments: loader.efi
   Image base: 0x7e8d7000
   EFI version: 2.70
   EFI Firmware: BHYVE (rev 1.00)
   Console: efi (0x1000)
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-84-87-20-FC-9C-58)/HD(1,GPT,92F393EA-5F1C-11EC-AC0F-002590F15508,0x28,0x82000)
   BootCurrent: 0001
   BootOrder: 0000 0001[*] 0002
   BootInfo Path: PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-84-87-20-FC-9C-58)
Ignoring Boot0001: Only one DP found
Trying ESP: PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-84-87-20-FC-9C-58)/HD(1,GPT,92F393EA-5F1C-11EC-AC0F-002590F15508,0x28,0x82000)
Setting currdev to disk0p1:
Trying: PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-84-87-20-FC-9C-58)/HD(2,GPT,930B40B3-5F1C-11EC-AC0F-002590F15508,0x82800,0x800000)
Setting currdev to disk0p2:
Trying: PciRoot(0x0)/Pci(0x4,0x0)/NVMe(0x1,01-00-84-87-20-FC-9C-58)/HD(3,GPT,93204892-5F1C-11EC-AC0F-002590F15508,0x882800,0x297D000)
Setting currdev to zfs:rpool/ROOT/default:
\

Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hints
Loading /boot/loader.conf
Loading /boot/loader.conf.local
Loading /boot/loader.conf.d/carp.conf
Loading /boot/loader.conf.d/autoboot.conf
\
?c|
-  ______               ____   _____ _____
  |  ____|             |  _ \ / ____|  __ \
  | |___ _ __ ___  ___ | |_) | (___ | |  | |
  |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
  | |   | | |  __/  __/| |_) |____) | |__| |
  | |   | | |    |    ||     |      |      |
  |_|   |_|  \___|\___||____/|_____/|_____/      ```                        `
                                                s` `.....---.......--.```   -/
 ������������Welcome to FreeBSD�������������    +o   .--`         /y:`      +.
 �                                         �     yo`:.            :o      `+-
 �  1. Boot Multi user [Enter]             �      y/               -/`   -o/
 �  2. Boot Single user                    �     .-                  ::/sy+:.
 �  3. Escape to loader prompt             �     /                     `--  /
 �  4. Reboot                              �    `:                          :`
 �  5. Cons: Serial                        �    `:                          :`
 �                                         �     /                          /
 �  Options:                               �     .-                        -.
 �  6. Kernel: default/kernel (1 of 2)     �      --                      -.
 �  7. Boot Options                        �       `:`                  `:`
 �  8. Boot Environments                   �         .--             `--.
 �                                         �            .---.....----.
 �������������������������������������������
   Autoboot in 0 seconds, hit [Enter] to boot or any other key to stop

Loading kernel...
/boot/kernel/kernel text=0x17b9e0 text=0xdd6d60 text=0x65b9dc data=0x140 data=0x1b9348+0x445cb8 syms=[0x8+0x178ea8+0x8+0x19906f]
Loading configured modules...
/boot/kernel/carp.ko size 0x10c28 at 0x2112000
/boot/entropy size=0x1000
/boot/kernel/cryptodev.ko size 0xae38 at 0x2124000
/etc/hostid size=0x25
/boot/kernel/zfs.ko size 0x67feb0 at 0x212f000
Start @ 0xffffffff8037c000 ...
EFI framebuffer information:
addr, size     0x0, 0x0
dimensions     0 x 0
stride         0
masks          0x00000000, 0x00000000, 0x00000000, 0x00000000
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021
    root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (2101.68-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0x9f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT,PBE>
  Features2=0xfe9e6217<SSE3,PCLMULQDQ,DTES64,DS_CPL,SSSE3,CX16,xTPR,PCID,DCA,SSE4.1,SSE4.2,POPCNT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x24100800<SYSCALL,NX,Page1GB,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  Structured Extended Features3=0x400<MD_CLEAR>
  XSAVE Features=0x1<XSAVEOPT>
  TSC: P-state invariant
Hypervisor: Origin = "bhyve bhyve " 
real memory  = 2147483648 (2048 MB)
avail memory = 2040246272 (1945 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <BHYVE  BVMADT  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 hardware threads
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG" 
random: unblocking device.
ioapic0: MADT APIC ID 4 != hw id 0
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-31
Launching APs: 3 2 1
Timecounter "TSC" frequency 2101679808 Hz quality 1000
KTLS: Initialized 4 threads
random: entropy device external interface
kernel trap 12 with interrupts disabled

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fault virtual address    = 0x0
fault code        = supervisor read data, page not present
instruction pointer    = 0x20:0xffffffff80be8671
stack pointer            = 0x28:0xfffffe005133eb90
frame pointer            = 0x28:k0exfrfnffefle0 0t51r3a3epba 01
tyo dwe isetghme nit    n    t= ebrasre u0px0t,s l imdiit s0xafbfflfef,d
  pe
10x
Fb
     a        t=a DlP L t0r, appre s1 12,: l onpga 1g,e d eff3a2u 0l, tg rawn h1i
Ipero ceisns ork eefrlangse    l=  rmesoumdee,
 OPcL p=u 0i
)c ur=re n3t ;pr oacepsis    c    =  i13d ( g=_e v0en3t

tfraapu nlutmb erv    i    =r 1t2u
apanic: page fault
cpuid = 2
time = 1
KDB: stack backtrace:
#0 0xffffffff80c574c5 at kdb_backtrace+0x65
#1 0xffffffff80c09ea1 at vpanic+0x181
#2 0xffffffff80c09d13 at panic+0x43
#3 0xffffffff8108b1b7 at trap_fatal+0x387
#4 0xffffffff8108b20f at trap_pfault+0x4f
#5 0xffffffff8108a86d at trap+0x27d
#6 0xffffffff81061958 at calltrap+0x8
#7 0xffffffff80b4f1e0 at g_event_procbody+0x10
#8 0xffffffff80bc7dde at fork_exit+0x7e
#9 0xffffffff810629de at fork_trampoline+0xe
Uptime: 1s
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 2

Not sure if relevant but it looks like both FreeBSD and my Linux one are all trying to access address 0x0, just something I noticed.

There are commits around that time frame it started happening:
https://github.com/illumos/illumos-gate/commit/d1c02647196254b902261506af8a31be3080ca68
https://github.com/illumos/illumos-gate/commit/8779b44894428ca4ea78d1bfe5381291f62e9c7d
https://github.com/illumos/illumos-gate/commit/0153d828c132fdb1a17c11b99386a3d1b87994cf

I don't think I've seen it on my single socket system only on my multi-socket system (ivy bridge)

As requested on IRC, some info on the physical machine:

root@saturn:~# prtdiag
System Configuration: Supermicro X9DRi-LN4+/X9DR3-LN4+
BIOS Configuration: American Megatrends Inc. 3.2 03/04/2015
BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style)

==== Processor Sockets ====================================

Version                          Location Tag
-------------------------------- --------------------------
Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz CPU 1
Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz CPU 2

==== Memory Device Sockets ================================

Type        Status Set Device Locator      Bank Locator
----------- ------ --- ------------------- ----------------
DDR3        in use 0   P1-DIMMA1           P0_Node0_Channel0_Dimm0
DDR3        in use 0   P1-DIMMA2           P0_Node0_Channel0_Dimm1
unknown     empty  0   P1-DIMMA3           P0_Node0_Channel0_Dimm2
DDR3        in use 0   P1-DIMMB1           P0_Node0_Channel1_Dimm0
DDR3        in use 0   P1-DIMMB2           P0_Node0_Channel1_Dimm1
unknown     empty  0   P1-DIMMB3           P0_Node0_Channel1_Dimm2
unknown     empty  0   P1-DIMMC1           P0_Node0_Channel2_Dimm0
unknown     empty  0   P1-DIMMC2           P0_Node0_Channel2_Dimm1
unknown     empty  0   P1-DIMMC3           P0_Node0_Channel2_Dimm2
unknown     empty  0   P1-DIMMD1           P0_Node0_Channel3_Dimm0
unknown     empty  0   P1-DIMMD2           P0_Node0_Channel3_Dimm1
unknown     empty  0   P1-DIMMD3           P0_Node0_Channel3_Dimm2
DDR3        in use 0   P2-DIMME1           P1_Node1_Channel0_Dimm0
DDR3        in use 0   P2-DIMME2           P1_Node1_Channel0_Dimm1
unknown     empty  0   P2-DIMME3           P1_Node1_Channel0_Dimm2
DDR3        in use 0   P2-DIMMF1           P1_Node1_Channel1_Dimm0
DDR3        in use 0   P2-DIMMF2           P1_Node1_Channel1_Dimm1
unknown     empty  0   P2-DIMMF3           P1_Node1_Channel1_Dimm2
unknown     empty  0   P2-DIMMG1           P1_Node1_Channel2_Dimm0
unknown     empty  0   P2-DIMMG2           P1_Node1_Channel2_Dimm1
unknown     empty  0   P2-DIMMG3           P1_Node1_Channel2_Dimm2
unknown     empty  0   P2-DIMMH1           P1_Node1_Channel3_Dimm0
unknown     empty  0   P2-DIMMH2           P1_Node1_Channel3_Dimm1
unknown     empty  0   P2-DIMMH3           P1_Node1_Channel3_Dimm2

==== On-Board Devices =====================================
 Onboard Matrox VGA
 Onboard Intel Ethernet 1
 Onboard Intel Ethernet 2
 Onboard Intel Ethernet 3
 Onboard Intel Ethernet 4

==== Upgradeable Slots ====================================

ID  Status    Type             Description
--- --------- ---------------- ----------------------------
1   available PCI Exp. Gen 3 x16 CPU1 SLOT1 PCI-E 3.0 X16
2   in use    PCI Exp. Gen 3 x4 CPU1 SLOT2 PCI-E 3.0 X4 (IN X8 SLOT), pci12d8,8608 (pcieb)
3   available PCI Exp. Gen 3 x16 CPU1 SLOT3 PCI-E 3.0 X16
4   in use    PCI Exp. Gen 3 x16 CPU2 SLOT4 PCI-E 3.0 X16, Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (mpt_sas)
5   in use    PCI Exp. Gen 3 x16 CPU2 SLOT5 PCI-E 3.0 X16, Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] (mpt_sas)
6   in use    PCI Exp. Gen 3 x8 CPU2 SLOT6 PCI-E 3.0 X8, Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (ixgbe)
root@saturn:~# prtconf -m
131044
root@saturn:~# pptadm list
DEV        VENDOR DEVICE PATH
/dev/ppt0  1912   15     /pci@0,0/pci8086,e03@1,1/pci12d8,8608@0/pci12d8,8608@1/pci1912,15@0
/dev/ppt1  1912   15     /pci@0,0/pci8086,e03@1,1/pci12d8,8608@0/pci12d8,8608@5/pci1912,15@0
/dev/ppt2  1912   15     /pci@0,0/pci8086,e03@1,1/pci12d8,8608@0/pci12d8,8608@7/pci1912,15@0
/dev/ppt3  1912   15     /pci@0,0/pci8086,e03@1,1/pci12d8,8608@0/pci12d8,8608@9/pci1912,15@0
/dev/ppt4  8086   1521   /pci@0,0/pci8086,1d10@1c/pci15d9,1521@0
/dev/ppt5  8086   1521   /pci@0,0/pci8086,1d10@1c/pci15d9,1521@0,1
/dev/ppt6  8086   1521   /pci@0,0/pci8086,1d10@1c/pci15d9,1521@0,2
/dev/ppt7  8086   1521   /pci@0,0/pci8086,1d10@1c/pci15d9,1521@0,3
root@saturn:~#

So it's an Ivy Bridge EP with 128G of memory, nothing too fancy. I have a PCIe card with 4 xhci controller I use for passthru as well as 4x onboard igb.

As for the VM it's mostly ares that is hitting it (unrar/par2 workloads or ffmpeg transcoding), artemis (ffmpeg transcoding), very rarely proteus or rcon2 during boot.

ares: Ubuntu 21.10 (GNU/Linux 5.13.0-25-generic x86_64)
artemis: Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-94-generic x86_64)
proteus: FreeBSD 13.0-RELEASE-p6
rcon2: FreeBSD 13.0-RELEASE-p6

root@saturn:~# zadm show ares
{
   "acpi" : "on",
   "autoboot" : "true",
   "bootargs" : "",
   "bootdisk" : {
      "blocksize" : "4K",
      "path" : "rpool/vms/ares/disk0",
      "size" : "25G",
      "sparse" : "false" 
   },
   "bootorder" : "cd",
   "bootrom" : "BHYVE_RELEASE-202111",
   "brand" : "bhyve",
   "cloud-init" : "off",
   "cpu-shares" : "30",
   "dataset" : [
      {
         "name" : "rpool/vmdata/ares" 
      }
   ],
   "diskif" : "nvme",
   "fs-allowed" : "",
   "hostbridge" : "i440fx",
   "hostid" : "",
   "ip-type" : "exclusive",
   "limitpriv" : "default",
   "net" : [
      {
         "global-nic" : "ixgbe0",
         "mac-addr" : "00:22:06:b5:f5:2d",
         "physical" : "ares0",
         "vlan-id" : "100",
         "vqsize" : "32768" 
      }
   ],
   "netif" : "virtio",
   "pool" : "",
   "ram" : "4G",
   "rng" : "off",
   "scheduling-class" : "",
   "type" : "generic",
   "vcpus" : "4,sockets=1,cores=2,threads=2",
   "vga" : "off",
   "virtfs" : [
      {
         "name" : "wwwdata",
         "path" : "/vol/www",
         "ro" : "false" 
      },
      {
         "name" : "httpd",
         "path" : "/vol/httpd",
         "ro" : "false" 
      },
      {
         "name" : "radarr",
         "path" : "/vol/radarr",
         "ro" : "false" 
      },
      {
         "name" : "lidarr",
         "path" : "/vol/lidarr",
         "ro" : "false" 
      },
      {
         "name" : "sonarr",
         "path" : "/vol/sonarr",
         "ro" : "false" 
      },
      {
         "name" : "transmission",
         "path" : "/vol/transmission",
         "ro" : "false" 
      }
   ],
   "vnc" : "off",
   "xhci" : "off",
   "zonename" : "ares",
   "zonepath" : "/zones/ares" 
}
root@saturn:~# zadm show artemis
{
   "acpi" : "on",
   "autoboot" : "true",
   "bootargs" : "",
   "bootdisk" : {
      "blocksize" : "4K",
      "path" : "rpool/vms/artemis/disk0",
      "size" : "35G",
      "sparse" : "false" 
   },
   "bootorder" : "cd",
   "bootrom" : "BHYVE_RELEASE-202111",
   "brand" : "bhyve",
   "cloud-init" : "off",
   "cpu-shares" : "25",
   "dataset" : [
      {
         "name" : "rpool/vmdata/artemis" 
      }
   ],
   "diskif" : "nvme",
   "fs-allowed" : "",
   "hostbridge" : "i440fx",
   "hostid" : "",
   "ip-type" : "exclusive",
   "limitpriv" : "default",
   "net" : [
      {
         "global-nic" : "ixgbe0",
         "mac-addr" : "00:22:06:63:25:df",
         "physical" : "artemis0",
         "vlan-id" : "100",
         "vqsize" : "16384" 
      }
   ],
   "netif" : "virtio",
   "pool" : "",
   "ram" : "8G",
   "rng" : "off",
   "scheduling-class" : "",
   "type" : "generic",
   "vcpus" : "12,sockets=1,cores=6,threads=2",
   "vga" : "off",
   "virtfs" : [
      {
         "name" : "plexdata",
         "path" : "/vol/plexdata",
         "ro" : "false" 
      }
   ],
   "vnc" : "off",
   "xhci" : "off",
   "zonename" : "artemis",
   "zonepath" : "/zones/artemis" 
}
root@saturn:~# zadm show rcon2
{
   "acpi" : "on",
   "autoboot" : "true",
   "bootargs" : "",
   "bootdisk" : {
      "blocksize" : "4K",
      "path" : "rpool/vms/rcon2/disk0",
      "size" : "10G",
      "sparse" : "false" 
   },
   "bootorder" : "cd",
   "bootrom" : "BHYVE_RELEASE-202111",
   "brand" : "bhyve",
   "cloud-init" : "off",
   "cpu-shares" : "25",
   "diskif" : "nvme",
   "fs-allowed" : "",
   "hostbridge" : "i440fx",
   "hostid" : "",
   "ip-type" : "exclusive",
   "limitpriv" : "default",
   "netif" : "virtio",
   "pool" : "",
   "ppt" : [
      {
         "device" : "ppt0",
         "state" : "on" 
      },
      {
         "device" : "ppt1",
         "state" : "on" 
      },
      {
         "device" : "ppt5",
         "state" : "on" 
      }
   ],
   "ram" : "1G",
   "rng" : "off",
   "scheduling-class" : "",
   "type" : "generic",
   "vcpus" : "1,sockets=1,cores=1,threads=1",
   "vga" : "off",
   "vnc" : "off",
   "xhci" : "off",
   "zonename" : "rcon2",
   "zonepath" : "/zones/rcon2" 
}
root@saturn:~# zadm show proteus
{
   "acpi" : "on",
   "autoboot" : "true",
   "bootargs" : "",
   "bootdisk" : {
      "blocksize" : "8K",
      "path" : "rpool/vms/proteus/disk0",
      "size" : "25G",
      "sparse" : "true" 
   },
   "bootorder" : "cd",
   "bootrom" : "BHYVE_RELEASE-202111",
   "brand" : "bhyve",
   "cloud-init" : "off",
   "cpu-shares" : "25",
   "diskif" : "nvme",
   "fs-allowed" : "",
   "hostbridge" : "i440fx",
   "hostid" : "",
   "ip-type" : "exclusive",
   "limitpriv" : "default",
   "netif" : "virtio",
   "pool" : "",
   "ppt" : [
      {
         "device" : "ppt4",
         "state" : "on" 
      }
   ],
   "ram" : "2G",
   "rng" : "off",
   "scheduling-class" : "",
   "type" : "generic",
   "vcpus" : "4,sockets=1,cores=2,threads=2",
   "vga" : "off",
   "vnc" : "off",
   "xhci" : "off",
   "zonename" : "proteus",
   "zonepath" : "/zones/proteus" 
}

The following is able to trigger it for me, but not consisteantly:

yt-dlp -c --merge-output-format mp4 --format bestvideo+bestaudio  --compat-options all,-no-live-chat --add-metadata --embed-subs --all-subs --convert-subs ass https://www.youtube.com/watch?v=IdRagVTlrmo

I had it die once in the rar command about 10% in, but the 2nd attempt I manage todo everything so also not really reliable.

rar -v256000k a vmdump.rar vmdump.3
par2 create -m1024 -r10 vmdump.partX.rar vmdump.*.rar
# note: damage some rar files
dd if=/dev/urandom count=8 of=vmdump.part13.rar conv=notrunc
par2verify vmdump.partX.rar.par2
par2repair vmdump.partX.rar.par2

Another one, this one got stuck as the linux kernel failed to reboot, including console output, vmm stack, pstack, ...
https://gist.github.com/sjorge/34165dd781a4646ab7b71d2422444443

After running on a build with 13896 reverted, the vm just died instead and bhyve logged an EPT violation

[23:34:52] <sjorge> rdmsr to register 0x64d on vcpu 0
[23:34:52] <sjorge> vm exit[3]
[23:34:52] <sjorge> reason          VMX
[23:34:52] <sjorge> rip             0x00000000006f0c20
[23:34:52] <sjorge> inst_length     5
[23:34:52] <sjorge> status          0
[23:34:52] <sjorge> exit_reason     48 (EPT violation)
[23:34:52] <sjorge> qualification   0x0000000000000184
[23:34:52] <sjorge> inst_type               0
[23:34:52] <sjorge> inst_error              0

All this was on a patchset with all the memory related commits from the past few months reverted, I will update to tonights bloody + patchset from andy to see if it also stays OK on the newer bits.

[   14.200961] fbcon: Taking over console
 09:24:06 up 1 day, 58 min,  0 users,  load average: 0.07, 0.03, 0.00

We made it past 24h without any issues! But...

This is with Hyper Threading disabled in the BIOS, it was enabled before. I noticed I was seeing more cores than expected while collecting some info, I had HT disabled in the past given this is older gen CPU and well spectr/meltdown and friends happened.

This has also brought some other improvements, these messages are now gone on all my VMs

[11:50:41] <sjorge> [  463.930275] hrtimer: interrupt took 147684029 ns
[11:50:42] <sjorge> [  570.011699] clocksource: timekeeping watchdog on CPU2: hpet read-back delay of 1910269ns, attempt 4, marking unstable
[11:50:42] <sjorge> [  570.788413] tsc: Marking TSC unstable due to clocksource watchdog
[11:50:42] <sjorge> [  570.788977] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
[11:50:42] <sjorge> [  570.788981] sched_clock: Marking unstable (570576256076, 210875811)<-(570795670587, -6693491)
[11:50:42] <sjorge> [  570.787375] clocksource: Checking clocksource tsc synchronization from CPU 2.
[11:50:42] <sjorge> [  570.787516] clocksource: Switched to clocksource hpet

Eventually it would also kick out HPET and switch to PIT for timings in the VM. So it seems Hyper Threads are at lease causing some kind of unstable clock for the VM and it's not happy, I'm not sure but there might be a relation with the page faults?

Less interesting blurp:
The above mentioned EPT Violation was hit once after the system booted, after enabling the debug flag to not take down the zone the VM did hang a few more times before I disabled HT (not after), but it would eventually reset the guest without the EPT violation showing up in the logs, the console had the usual strack traces (I had already remove panic=15 from the grub linux params)

I don't think it ever got into the state pmooney was interested in (dead vm with ept violation in log).

I'll update once todays bloody get dropped and andy provides a patch to see if the its also stable wit HT disabled.

Currently on OmniOS Bloody + hf (with bhyve sync bits) and the VM didn't make it past 24h.

Same issue, HT still disable in the Host BIOS

[   19.156735] BUG: unable to handle page fault for address: ffff964f96982000
[   19.159022] #PF: supervisor write access in kernel mode                                                            [   19.160761] #PF: error_code(0x0003) - permissions violation
[   19.162711] PGD 32801067 P4D 32801067 PUD 10030c063 PMD 10ef4c063 PTE 8000000116982161
[   19.165501] Oops: 0003 [#1] SMP PTI
[   19.166755] CPU: 1 PID: 566 Comm: cifsd Not tainted 5.13.0-27-generic #29-Ubuntu
[   19.169299] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   19.171609] RIP: 0010:memcpy_erms+0x6/0x10
[   19.173043] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1
f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   19.179376] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   19.181044] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   19.183300] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   19.185740] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80
[   19.188187] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   19.190616] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800
[   19.193055] FS:  0000000000000000(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   19.195815] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.197780] CR2: ffff964f96982000 CR3: 0000000116498004 CR4: 00000000001706e0
[   19.200215] Call Trace:
[   19.201086]  ? receive_encrypted_standard+0xf9/0x310 [cifs]
[   19.203103]  smb3_receive_transform+0x4d/0xf0 [cifs]
[   19.204878]  cifs_demultiplex_thread+0x73c/0x810 [cifs]
[   19.206730]  ? __schedule+0x270/0x680
[   19.208024]  kthread+0x11f/0x140
[   19.209155]  ? cifs_handle_standard+0x1a0/0x1a0 [cifs]
[   19.210890]  ? set_kthread_struct+0x50/0x50
[   19.212340]  ret_from_fork+0x22/0x30
[   19.213598] Modules linked in: md4 cmac nls_utf8 cifs libarc4 libdes 9p fscache netfs binfmt_misc nls_iso8859_1 int
el_rapl_msr intel_rapl_common rapl input_leds serio_raw 9pnet_virtio mac_hid 9pnet efi_pstore sch_fq_codel msr drm sun
rpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async
_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni
_intel aesni_intel crypto_simd nvme cryptd virtio_net net_failover psmouse failover nvme_core
[   19.230316] CR2: ffff964f96982000
[   19.231482] ---[ end trace fd6ac84d0e480290 ]---                                                                   [   19.231484] BUG: unable to handle page fault for address: 0000000002fffff6
[   19.235114] #PF: supervisor read access in kernel mode
[   19.236605] #PF: error_code(0x0000) - not-present page
[   19.238094] PGD 0 P4D 0
[   19.238849] Oops: 0000 [#2] SMP PTI
[   19.239876] CPU: 0 PID: 553 Comm: Radarr Tainted: G      D           5.13.0-27-generic #29-Ubuntu
[   19.241025] RIP: 0010:memcpy_erms+0x6/0x10
[   19.242429] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   19.243866] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1
f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   19.245795] RIP: 0010:vma_dump_size+0x2c/0x170
[   19.252236] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   19.253543] Code: 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 fb 48 8b 7f 40 e8 55 5c c4 ff 48 39 c3 74 22 48 8b 8
3 90 00 00 00 48 85 c0 74 22 <48> 8b 40 60 48 85 c0 74 19 48 89 df e8 73 29 c4 00 48 85 c0 74 0c
[   19.255149] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   19.260492] RSP: 0018:ffffb651006dfa30 EFLAGS: 00010206
[   19.262642] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   19.264149] RAX: 0000000002ffff96 RBX: ffff964f96980e10 RCX: ffffb651006dfaf0                                      [   19.266299] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80                                      [   19.268346] RDX: ffff964f84030000 RSI: 00000000000000cd RDI: 0000000000000000                                      [   19.270494] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80                                      [   19.272542] RBP: ffffb651006dfa40 R08: 0000000000033240 R09: 0000000000000006
[   19.274941] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800
[   19.277000] R10: 0000000000001460 R11: ffffed18c4100c00 R12: 00000000000000cd
[   19.279175] FS:  0000000000000000(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000                           [   19.281227] R13: ffff964f96980e10 R14: ffffb651006dfd48 R15: 0000000000000283
[   19.283885] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.285936] FS:  00007fc25259f740(0000) GS:ffff964fbbc00000(0000) knlGS:0000000000000000
[   19.287913] CR2: ffff964f96982000 CR3: 0000000116498004 CR4: 00000000001706e0
[   19.290228] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.294351] CR2: 0000000002fffff6 CR3: 0000000105f66002 CR4: 00000000001706f0
[   19.296567] Call Trace:
[   19.297309]  dump_vma_snapshot+0x121/0x230
[   19.298504]  elf_core_dump+0x60/0x560
[   19.299584]  ? memcg_slab_post_alloc_hook+0x48/0x230
[   19.301027]  ? __kmalloc_node+0x169/0x300
[   19.302200]  do_coredump+0x5a8/0x8a0
[   19.303249]  ? kmem_cache_free+0xff/0x430
[   19.304422]  ? refcount_dec_and_lock_irqsave+0x1c/0xa0
[   19.305911]  ? __sigqueue_free.part.0+0x27/0x40
[   19.307232]  get_signal+0x14b/0x8b0
[   19.308257]  arch_do_signal_or_restart+0xd9/0x100
[   19.309620]  ? kill_something_info+0x175/0x1a0
[   19.310917]  exit_to_user_mode_loop+0xc4/0x160
[   19.312213]  exit_to_user_mode_prepare+0x9f/0xb0
[   19.313558]  syscall_exit_to_user_mode+0x27/0x50
[   19.314906]  do_syscall_64+0x6e/0xb0
[   19.315968]  ? syscall_exit_to_user_mode+0x27/0x50
[   19.317364]  ? __do_sys_gettid+0x1b/0x20
[   19.318511]  ? do_syscall_64+0x6e/0xb0
[   19.319613]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[   19.321081] RIP: 0033:0x7fc25263e808
[   19.322132] Code: 00 48 81 c4 90 00 00 00 44 89 e8 5b 5d 41 5c 41 5d 41 5e c3 66 0f 1f 44 00 00 b8 ba 00 00 00 0f 0
5 89 c7 b8 3e 00 00 00 0f 05 <41> 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00 44 0f 46 e8 eb aa
[   19.327466] RSP: 002b:00007ffe21b80650 EFLAGS: 00000246 ORIG_RAX: 000000000000003e
[   19.329640] RAX: 0000000000000000 RBX: 00007fc25259f740 RCX: 00007fc25263e808
[   19.331693] RDX: 00000000ffffffe6 RSI: 0000000000000006 RDI: 0000000000000229
[   19.333750] RBP: 0000000000000006 R08: 00007ffe21b80720 R09: 0000000000000000                                      [   19.335815] R10: 0000000000000008 R11: 0000000000000246 R12: 000055b65b804f8a
[   19.337867] R13: 0000000000000016 R14: 0000000000000001 R15: 0000000000000000                            [696/1637]
[   19.339919] Modules linked in: md4 cmac nls_utf8 cifs libarc4 libdes 9p fscache netfs binfmt_misc nls_iso8859_1 int
el_rapl_msr intel_rapl_common rapl input_leds serio_raw 9pnet_virtio mac_hid 9pnet efi_pstore sch_fq_codel msr drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async
_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni
_intel aesni_intel crypto_simd nvme cryptd virtio_net net_failover psmouse failover nvme_core
[   19.353898] CR2: 0000000002fffff6
[   19.354876] general protection fault, probably for non-canonical address 0xced28d51f2a949dc: 0000 [#3] SMP PTI
[   19.354985] ---[ end trace fd6ac84d0e480291 ]---
[   19.358296] CPU: 1 PID: 27 Comm: kworker/1:1 Tainted: G      D           5.13.0-27-generic #29-Ubuntu
[   19.358301] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   19.358303] Workqueue: deferredclose smb2_deferred_work_close [cifs]
[   19.367115] RIP: 0010:kmem_cache_alloc_node+0xac/0x250
[   19.367316] RIP: 0010:memcpy_erms+0x6/0x10
[   19.368688] Code: 48 85 c9 0f 84 80 01 00 00 8b 5d bc 83 fb ff 74 0f 48 8b 09 48 c1 e9 36 39 cb 0f 85 69 01 00 00 4
1 8b 4e 28 49 8b 3e 48 01 c1 <48> 8b 19 48 89 ce 49 33 9e b8 00 00 00 48 8d 4a 01 48 0f ce 48 31
[   19.369896] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe                      [   19.375949] RSP: 0018:ffffb651000fb6a8 EFLAGS: 00010282                                                            [   19.375953] RAX: ced28d51f2a948fc RBX: 00000000ffffffff RCX: ced28d51f2a949dc                                      [   19.375955] RDX: 00000000000012d5 RSI: 0000000000000c40 RDI: 0000000000034aa0
[   19.375957] RBP: ffffb651000fb6f0 R08: ffff964fbbd34aa0 R09: ffff964f8e31e6e0
[   19.375958] R10: 0000000000000000 R11: 0000000000000010 R12: 0000000000000c40
[   19.375960] R13: ffff964f80302e00 R14: ffff964f80302e00 R15: ffffffffa6bd22ad                                      [   19.375962] FS:  0000000000000000(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   19.381328] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   19.383108] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.385181]
[   19.387595] CR2: 00007fa0c24f1d38 CR3: 0000000116d2a002 CR4: 00000000001706e0
[   19.387598] Call Trace:
[   19.387604]  __alloc_skb+0x15d/0x1d0
[   19.389650] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   19.392081]  sk_stream_alloc_skb+0x6e/0x2e0
[   19.392090]  tcp_sendmsg_locked+0x598/0xc10
[   19.394143] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   19.396885]  tcp_sendmsg+0x2d/0x50
[   19.396891]  inet6_sendmsg+0x43/0x70
[   19.398407] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80
[   19.400387]  sock_sendmsg+0x48/0x70
[   19.400393]  smb_send_kvec+0x88/0x170 [cifs]
[   19.400856] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   19.403299]  __smb_send_rqst+0x4a6/0x690 [cifs]
[   19.404035] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800
[   19.405293]  smb_send_rqst+0x131/0x1a0 [cifs]
[   19.407351] FS:  00007fc25259f740(0000) GS:ffff964fbbc00000(0000) knlGS:0000000000000000
[   19.408784]  compound_send_recv+0x35e/0x970 [cifs]
[   19.409999] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.412440]  ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
[   19.412449]  cifs_send_recv+0x22/0x30 [cifs]
[   19.413446] CR2: 0000000002fffff6 CR3: 0000000105f66002 CR4: 00000000001706f0
[   19.414637]  __SMB2_close+0x25d/0x420 [cifs]
[   19.416774] Lidarr invoked oom-killer: gfp_mask=0x0(), order=0, oom_score_adj=0
[   19.417784]  ? ttwu_do_wakeup+0x1c/0x140
[   19.419059] CPU: 0 PID: 612 Comm: Lidarr Tainted: G      D           5.13.0-27-generic #29-Ubuntu
[   19.421215]  smb2_close_getattr+0x33/0x110 [cifs]
[   19.422548] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   19.424988]  ? smb2_close_getattr+0x33/0x110 [cifs]                                                                [   19.426268] Call Trace:
[   19.429025]  _cifsFileInfo_put+0x2b6/0x3b0 [cifs]
[   19.430419]  show_stack+0x52/0x58
[   19.432393]  ? cifs_del_deferred_close+0x79/0x80 [cifs]                                                            [   19.434285]  dump_stack+0x7d/0x9c
[   19.435765]  ? __raw_callee_save___native_queued_spin_unlock+0x15/0x23
[   19.437827]  dump_header+0x4f/0x1f9
[   19.439305]  smb2_deferred_work_close+0x61/0x70 [cifs]
[   19.441416]  oom_kill_process.cold+0xb/0x10
[   19.442623]  process_one_work+0x220/0x3c0
[   19.445184]  out_of_memory.part.0+0xce/0x270
[   19.446732]  worker_thread+0x53/0x420
[   19.448678]  out_of_memory+0x41/0x80
[   19.450364]  kthread+0x11f/0x140
[   19.451099]  pagefault_out_of_memory+0x68/0x80
[   19.452718]  ? process_one_work+0x3c0/0x3c0
[   19.453702]  do_user_addr_fault+0x4e5/0x660
[   19.455522]  ? set_kthread_struct+0x50/0x50                                                                        [   19.456520]  ? exit_to_user_mode_prepare+0x37/0xb0                                                                 [   19.458678]  ret_from_fork+0x22/0x30                                                                               [   19.459711]  exc_page_fault+0x7d/0x170                                                                             [   19.461475] Modules linked in:
[   19.462710]  ? asm_exc_page_fault+0x8/0x30
[   19.464097]  md4
[   19.465352]  asm_exc_page_fault+0x1e/0x30                                                                          [   19.466621]  cmac
[   19.467676] RIP: 0033:0x7fcf9db8d470
[   19.468800]  nls_utf8
[   19.470107] Code: ff 49 39 c5 0f 84 6e 01 00 00 41 8d 47 01 89 45 88 bb ff ff ff ff 41 bc 01 00 00 00 89 5d 8c 4c 8
9 eb 0f 1f 84 00 00 00 00 00 <4c> 8b 2b 8b 0d 87 95 5e 00 44 39 f9 8b 43 10 76 0f 39 c8 73 1c 44
[   19.471547]  cifs
[   19.472774] RSP: 002b:00007fcf9ad26c80 EFLAGS: 00010283
[   19.474215]  libarc4
[   19.475612]
[   19.476851]  libdes
[   19.477955] RAX: 0000000000004b79 RBX: 0000000002ba5fc0 RCX: 00007fcf9ad26c90
[   19.479020]  9p
[   19.480221] RDX: 0000000000000000 RSI: 00000000022b1c80 RDI: 0000000000000006
[   19.480889]  fscache
[   19.482064] RBP: 00007fcf9ad26d00 R08: 000000000000006e R09: 0000000000000004
[   19.482681]  netfs
[   19.483751] R10: 00007fff6852a080 R11: 0000000000000246 R12: 0000000000000001
[   19.484583]  binfmt_misc
[   19.489951] R13: 0000000002ba5fc0 R14: 00007fff6850dad8 R15: 0000000000004b78
[   19.490662]  nls_iso8859_1
[   19.492215] Mem-Info:
[   19.492891]  intel_rapl_msr
[   19.493361] active_anon:94 inactive_anon:45079 isolated_anon:0
[   19.493361]  active_file:17512 inactive_file:85772 isolated_file:0
[   19.493361]  unevictable:0 dirty:2200 writeback:0
[   19.493361]  slab_reclaimable:7309 slab_unreclaimable:6691
[   19.493361]  mapped:40142 shmem:169 pagetables:677 bounce:0
[   19.493361]  free:801000 free_pcp:1121 free_cma:0
[   19.494024]  intel_rapl_common
[   19.496148] Node 0 active_anon:376kB inactive_anon:180316kB active_file:70048kB inactive_file:343088kB unevictable:
0kB isolated(anon):0kB isolated(file):0kB mapped:160568kB dirty:8800kB writeback:0kB shmem:676kB shmem_thp: 0kB shmem_
pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:2680kB pagetables:2708kB all_unreclaimable? no
[   19.496698]  rapl                                                                                                  [   19.498765] Node 0
[   19.499541]  input_leds serio_raw 9pnet_virtio mac_hid 9pnet efi_pstore sch_fq_codel msr drm sunrpc ip_tables x_tab
les
[   19.501615] DMA free:14848kB min:260kB low:324kB high:388kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0kB boun
ce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[   19.502333]  autofs4
[   19.504410] lowmem_reserve[]:
[   19.505303]  btrfs
[   19.507381]  0
[   19.508331]  blake2b_generic
[   19.509014]  2888
[   19.509998]  zstd_compress
[   19.519228]  3841
[   19.520283]  raid10
[   19.528776]  3841
[   19.529455]  raid456
[   19.530087]  3841                                                                                                  [   19.533689]  async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul                                                                                             [   19.540704]                                                                                                        [   19.541469]  crc32_pclmul
[   19.542345] Node 0
[   19.543060]  ghash_clmulni_intel
[   19.543585] DMA32 free:3025504kB min:50624kB low:63280kB high:75936kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:3128688kB managed:3028816kB ml
ocked:0kB bounce:0kB free_pcp:2800kB local_pcp:1396kB free_cma:0kB
[   19.544580]  aesni_intel
[   19.545164] lowmem_reserve[]:
[   19.546113]  crypto_simd
[   19.546694]  0
[   19.547443]  nvme cryptd virtio_net net_failover psmouse failover nvme_core
[   19.547682] ---[ end trace fd6ac84d0e480292 ]---
[   19.548055]  0
[   19.559234] RIP: 0010:memcpy_erms+0x6/0x10
[   19.563401]  952 952
[   19.564305] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1
f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   19.565193]  952
[   19.566084] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   19.566595]
[   19.568981]
[   19.570323] Node 0
[   19.570925] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   19.572141] Normal free:163648kB min:16692kB low:20864kB high:25036kB reserved_highatomic:0KB active_anon:376kB ina
ctive_anon:180272kB active_file:70048kB inactive_file:343204kB unevictable:0kB writepending:8800kB present:1048576kB m
anaged:975436kB mlocked:0kB bounce:0kB free_pcp:1684kB local_pcp:600kB free_cma:0kB
[   19.572919] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   19.578269] lowmem_reserve[]: 0
[   19.578929] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80
[   19.580460]  0
[   19.581006] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   19.581469]  0
[   19.582215] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800
[   19.584289]  0 0
[   19.593320] FS:  0000000000000000(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   19.595384]
[   19.595386] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB                                                                  [   19.596485] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.598539] 0*64kB
[   19.599139] CR2: 00007fa0c24f1d38 CR3: 0000000116d2a002 CR4: 00000000001706e0
[   19.601210] 0*128kB 0*256kB 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 14848kB                               [   19.615242] Node 0 DMA32: 16*4kB (M) 10*8kB (M) 9*16kB (M) 14*32kB (M) 12*64kB (M) 11*128kB (M) 11*256kB (M) 12*512
kB (UM) 9*1024kB (UM) 9*2048kB (UM) 729*4096kB (M) = 3025504kB
[   19.617664] general protection fault, probably for non-canonical address 0x4f86a18b11ffffae: 0000 [#4] SMP PTI
[   19.619834] Node 0 Normal: 2*4kB (ME)
[   19.622877] CPU: 1 PID: 728 Comm: apport Tainted: G      D           5.13.0-27-generic #29-Ubuntu
[   19.622878] 5*8kB (UME) 13*16kB (UE)
[   19.626902] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   19.626907] RIP: 0010:vma_interval_tree_insert+0x37/0xb0
[   19.628000] 3*32kB
[   19.630283] Code: 4c 8d 5f 58 48 8b 47 08 48 2b 07 48 c1 e8 0c 4d 8d 44 01 ff 48 8b 06 48 89 e5 48 85 c0 74 5f 41 b
a 01 00 00 00 eb 03 48 89 d0 <4c> 39 40 18 73 04 4c 89 40 18 48 8d 48 10 4c 3b 48 40 72 07 48 8d
[   19.631837] (UE)
[   19.632491] RSP: 0018:ffffb65100acfc60 EFLAGS: 00010206
[   19.637852] 2*64kB                                                                                                 [   19.638526]                                                                                                        [   19.640062] (UE)                                                                                                   [   19.640706] RAX: 4f86a18b11ffff96 RBX: ffff964f81a93300 RCX: ffff964f96980388                                      [   19.641169] 3*128kB
[   19.641772] RDX: 4f86a18b11ffff96 RSI: ffff964f812db560 RDI: ffff964f954ac190
[   19.643834] (ME)
[   19.644506] RBP: ffffb65100acfc60 R08: 000000000000001b R09: 0000000000000000                                      [   19.646563] 2*256kB
[   19.647166] R10: 0000000000000000 R11: ffff964f954ac1e8 R12: ffff964f812db540
[   19.649239] (UM)
[   19.649907] R13: ffff964f8e3ddca8 R14: ffff964f8e3ddca0 R15: ffff964f954ac190
[   19.651981] 1*512kB
[   19.652641] FS:  0000000000000000(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   19.654704] (U)
[   19.655509] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.655511] CR2: 00007f8b9a63f270 CR3: 00000001094c8005 CR4: 00000000001706e0
[   19.655514] Call Trace:
[   19.655519]  __vma_link_file+0x46/0x50
[   19.657883] 2*1024kB (ME) 2*2048kB (UE)
[   19.658484]  vma_link+0x1a8/0x1d0
[   19.660169] 38*4096kB
[   19.662335]  mmap_region+0x4b7/0x6b0
[   19.663069] (UM)
[   19.664375]  do_mmap+0x485/0x5a0
[   19.664379]  vm_mmap_pgoff+0xd4/0x170
[   19.664385]  ksys_mmap_pgoff+0x1dc/0x260
[   19.665536] = 163680kB
[   19.666557]  __x64_sys_mmap+0x33/0x40
[   19.667277] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[   19.668374]  do_syscall_64+0x61/0xb0
[   19.668962] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[   19.670104]  ? exit_to_user_mode_prepare+0x37/0xb0
[   19.671191] 103492 total pagecache pages
[   19.672552]  ? syscall_exit_to_user_mode+0x27/0x50
[   19.672559]  ? __x64_sys_newfstatat+0x1c/0x20
[   19.673269] 0 pages in swap cache
[   19.674535]  ? do_syscall_64+0x6e/0xb0
[   19.677041] Swap cache stats: add 0, delete 0, find 0/0
[   19.678270]  ? do_syscall_64+0x6e/0xb0
[   19.680703] Free swap  = 2097148kB                                                                                 [   19.682334]  ? irqentry_exit_to_user_mode+0x9/0x20
[   19.683495] Total swap = 2097148kB
[   19.685131]  ? irqentry_exit+0x19/0x30
[   19.686403] 1048315 pages RAM                                                                                      [   19.687555]  ? exc_page_fault+0x8f/0x170

[   19.687561]  ? asm_exc_page_fault+0x8/0x30
[   19.687564]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[   19.688664] 0 pages HighMem/MovableOnly
[   19.690448] RIP: 0033:0x7f8b9a782386
[   19.691563] 43412 pages reserved
[   19.692702] Code: 5d 41 5c c3 f3 0f 1e fa 41 f7 c1 ff 0f 00 00 75 2b 55 48 89 fd 53 89 cb 48 85 ff 74 37 41 89 da 4
8 89 ef b8 09 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 5a 5b 5d c3 0f 1f 80 00 00 00 00 c7 05 46 1e
[   19.694094] 0 pages hwpoisoned
[   19.695139] RSP: 002b:00007ffe702d9138 EFLAGS: 00000246
[   19.696260] Tasks state (memory values in pages):
[   19.697194]  ORIG_RAX: 0000000000000009
[   19.698348] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[   19.699607] RAX: ffffffffffffffda RBX: 0000000000000802 RCX: 00007f8b9a782386                                      [   19.699610] RDX: 0000000000000001 RSI: 000000000001b0b8 RDI: 0000000000000000                                      [   19.699612] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000                                      [   19.699613] R10: 0000000000000802 R11: 0000000000000246 R12: 00007ffe702d9150                                      [   19.699614] R13: 0000000000000000 R14: 00007ffe702d9540 R15: 00007f8b9a753a20
[   19.699618] Modules linked in:
[   19.701111] [    361]     0   361    11511     3739    90112        0          -250 systemd-journal
[   19.702392]  md4                                                                                                   [   19.703455] [    396]     0   396     8429     2329    94208        0         -1000 systemd-udevd
[   19.704427]  cmac
[   19.709771] [    455]     0   455     3719     1270    69632        0             0 systemd-fsckd
[   19.710693]  nls_utf8
[   19.712220] [    486]   113   486     1983      995    57344        0             0 rpcbind
[   19.713769]  cifs
[   19.714890] [    487]   102   487    21916     1343    65536        0             0 systemd-timesyn
[   19.717789]  libarc4 libdes 9p fscache netfs binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common rapl input_
leds
[   19.719865] [    498]   100   498     4618     1714    73728        0             0 systemd-network
[   19.722266]  serio_raw
[   19.724339] [    500]   101   500     5878     2936    90112        0             0 systemd-resolve
[   19.726736]  9pnet_virtio
[   19.728887] [    523]   103   523     6171     3316    86016        0          -900 dbus-daemon
[   19.729832]  mac_hid
[   19.732514] [    530]     0   530    20641      942    61440        0             0 irqbalance
[   19.733154]  9pnet
[   19.735728] [    534]     0   534    10949     6876   122880        0             0 networkd-dispat
[   19.736358]  efi_pstore
[   19.738907] [    535]   104   535    59442     3598   114688        0             0 rsyslogd
[   19.739612]  sch_fq_codel msr drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid
456 async_raid6_recov
[   19.742020] [    538]     0   538     3304     1301    65536        0             0 systemd-logind
[   19.742619]  async_memcpy
[   19.745245] [    545]     0   545     6256     3401    90112        0         -1000 sshd
[   19.748430]  async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul
[   19.751054] [    547] 20001   547   788372    23084   487424        0             0 Lidarr
[   19.751786]  crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd nvme cryptd virtio_net net_failover psmouse f
ailover nvme_core
[   19.751917] ---[ end trace fd6ac84d0e480293 ]---
[   19.754404] [    553] 20001   553   820174    22688   483328        0             0 Radarr
[   19.765838] RIP: 0010:memcpy_erms+0x6/0x10
[   19.767261] [    554] 20001   554    93814    16441   278528        0             0 mono                           [   19.770909] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1
f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   19.773515] [    613]     0   613     5537     2808    90112        0             0 cron
[   19.774448] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282                                                            [   19.776802] [    627]     0   627     5269     2705    77824        0             0 agetty
[   19.780183]
[   19.782571] [    635]     0   635     4885     2618    77824        0             0 agetty
[   19.786213] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   19.787581] [    636] 20001   636    44076      800    98304        0             0 transmission-da
[   19.787585] [    640] 20001   640   297673     7014   249856        0             0 nzbget
[   19.787588] [    728]     0   728     1743      133    36864        0             0 apport
[   19.787591] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null)
[   19.790230] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   19.791455] ,cpuset=
[   19.794203] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80
[   19.799554] /
[   19.802310] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   19.803840] ,mems_allowed=0,global_oom                                                                             [   19.806647] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800                                      [   19.807110] ,task_memcg=                                                                                           [   19.809929] FS:  0000000000000000(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000                           [   19.812012] /system.slice/lidarr.service
[   19.815063] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.817460] ,task=Lidarr,pid=547,uid=20001
[   19.820274] CR2: 00007f8b9a63f270 CR3: 00000001094c8005 CR4: 00000000001706e0                                      [   19.822033] Out of memory: Killed process 547 (Lidarr) total-vm:3153488kB, anon-rss:34812kB, file-rss:57524kB, shme
m-rss:0kB, UID:20001 pgtables:476kB oom_score_adj:0
[   19.848320] BUG: unable to handle page fault for address: 0000000000002205
[   19.850352] #PF: supervisor read access in kernel mode
[   19.852168] #PF: error_code(0x0000) - not-present page
[   19.853766] PGD 800000010c8d7067 P4D 800000010c8d7067 PUD 0
[   19.855525] Oops: 0000 [#5] SMP PTI
[   19.856623] CPU: 0 PID: 554 Comm: mono Tainted: G      D           5.13.0-27-generic #29-Ubuntu
[   19.859221] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   19.861183] RIP: 0010:vma_dump_size+0x2c/0x170
[   19.862497] Code: 44 00 00 55 48 89 e5 41 54 49 89 f4 53 48 89 fb 48 8b 7f 40 e8 55 5c c4 ff 48 39 c3 74 22 48 8b 8
3 90 00 00 00 48 85 c0 74 22 <48> 8b 40 60 48 85 c0 74 19 48 89 df e8 73 29 c4 00 48 85 c0 74 0c
[   19.867889] RSP: 0018:ffffb651007c39c0 EFLAGS: 00010206
[   19.869418] RAX: 00000000000021a5 RBX: ffff964f96931d48 RCX: ffffb651007c3a80
[   19.871488] RDX: ffff964f85fa6000 RSI: 00000000000000cd RDI: 0000000000000000
[   19.873552] RBP: ffffb651007c39d0 R08: 0000000000002000 R09: ffff964f85fa6000
[   19.875623] R10: ffff964f803aac00 R11: ffff964f940ba198 R12: 00000000000000cd
[   19.877705] R13: ffff964f96931d48 R14: ffffb651007c3cd8 R15: 000000000000000d
[   19.879775] FS:  00007f7d2233c000(0000) GS:ffff964fbbc00000(0000) knlGS:0000000000000000
[   19.882112] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   19.883788] CR2: 0000000000002205 CR3: 0000000109950006 CR4: 00000000001706f0
[   19.883939] __swap_info_get: Bad swap offset entry 3690453cc3fff
[   19.885875] Call Trace:
[   19.885880]  dump_vma_snapshot+0x121/0x230
[   19.888278] BUG: Bad page map in process oom_reaper  pte:12df75867800000 pmd:116936067
[   19.888892]  elf_core_dump+0x60/0x560
[   19.890406] addr:0000000002a00000 vm_flags:08100073 anon_vma:ffff964f8c8cf840 mapping:0000000000000000 index:2a00
[   19.892721]  ? drain_obj_stock+0x32/0x90
[   19.892727]  ? __mod_memcg_state.part.0+0x2a/0x30
[   19.892731]  ? __mod_memcg_lruvec_state+0x27/0xe0
[   19.892735]  do_coredump+0x5a8/0x8a0
[   19.892737]  ? kmem_cache_free+0xff/0x430
[   19.892742]  ? refcount_dec_and_lock_irqsave+0x1c/0xa0                                                             [   19.892746]  ? __sigqueue_free.part.0+0x27/0x40
[   19.892752]  get_signal+0x14b/0x8b0
[   19.894098] file:(null) fault:0x0 mmap:0x0 readpage:0x0
[   19.897090]  arch_do_signal_or_restart+0xd9/0x100                                                                  [   19.897096]  ? kill_something_info+0x175/0x1a0
[   19.897100]  exit_to_user_mode_loop+0xc4/0x160
[   19.897104]  exit_to_user_mode_prepare+0x9f/0xb0
[   19.897106]  syscall_exit_to_user_mode+0x27/0x50
[   19.898688] CPU: 1 PID: 29 Comm: oom_reaper Tainted: G      D           5.13.0-27-generic #29-Ubuntu
[   19.899964]  do_syscall_64+0x6e/0xb0
[   19.901685] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   19.902749]  ? syscall_exit_to_user_mode+0x27/0x50
[   19.904218] Call Trace:
[   19.905721]  ? __do_sys_gettid+0x1b/0x20
[   19.907378]  show_stack+0x52/0x58
[   19.908418]  ? do_syscall_64+0x6e/0xb0
[   19.910338]  dump_stack+0x7d/0x9c
[   19.911733]  ? exit_to_user_mode_prepare+0x37/0xb0                                                                 [   19.913275]  print_bad_pte.cold+0x67/0xbf                                                                          [   19.914592]  ? syscall_exit_to_user_mode+0x27/0x50                                                                 [   19.916303]  zap_pte_range+0x6c5/0x790                                                                             [   19.917681]  ? do_syscall_64+0x6e/0xb0
[   19.920684]  zap_pmd_range.isra.0+0x1cc/0x2b0
[   19.921751]  ? exit_to_user_mode_prepare+0x37/0xb0
[   19.924154]  ? release_pages+0x163/0x4b0                                                                           [   19.925581]  ? syscall_exit_to_user_mode+0x27/0x50
[   19.926506]  unmap_page_range+0x218/0x3b0
[   19.927664]  ? do_syscall_64+0x6e/0xb0
[   19.928890]  __oom_reap_task_mm+0xf8/0x180
[   19.930004]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[   19.930009] RIP: 0033:0x7f7d223d7808
[   19.930013] Code: 00 48 81 c4 90 00 00 00 44 89 e8 5b 5d 41 5c 41 5d 41 5e c3 66 0f 1f 44 00 00 b8 ba 00 00 00 0f 0
5 89 c7 b8 3e 00 00 00 0f 05 <41> 89 c5 41 f7 dd 3d 00 f0 ff ff b8 00 00 00 00 44 0f 46 e8 eb aa
[   19.931241]  oom_reaper+0x274/0x350
[   19.932653] RSP: 002b:00007ffd6efbbe40 EFLAGS: 00000246
[   19.934125]  ? wait_woken+0x80/0x80
[   19.935531]  ORIG_RAX: 000000000000003e
[   19.936916]  kthread+0x11f/0x140
[   19.938027] RAX: 0000000000000000 RBX: 00007f7d2233c000 RCX: 00007f7d223d7808
[   19.939622]  ? __oom_reap_task_mm+0x180/0x180
[   19.941031] RDX: 00000000ffffffe6 RSI: 0000000000000006 RDI: 000000000000022a
[   19.942472]  ? set_kthread_struct+0x50/0x50
[   19.943881] RBP: 0000000000000006 R08: 00007ffd6efbbf10 R09: 0000000000000001
[   19.945350]  ret_from_fork+0x22/0x30
[   19.946466] R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000000
[   19.973712] R13: 0000000000000016 R14: 0000555fe5270000 R15: 0000000000000001
[   19.975806] Modules linked in: md4 cmac nls_utf8 cifs libarc4 libdes 9p fscache netfs binfmt_misc nls_iso8859_1 int
el_rapl_msr intel_rapl_common rapl input_leds serio_raw 9pnet_virtio mac_hid 9pnet efi_pstore sch_fq_codel msr drm sun
rpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async
_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni
_intel aesni_intel crypto_simd nvme cryptd virtio_net net_failover psmouse failover nvme_core
[   19.990149] CR2: 0000000000002205
[   19.991151] general protection fault, probably for non-canonical address 0xffa744c9c0000008: 0000 [#6] SMP PTI
[   19.991338] ---[ end trace fd6ac84d0e480294 ]---
[   19.994433] CPU: 1 PID: 612 Comm: Lidarr Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   19.998706] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.000894] RIP: 0010:acct_collect+0x70/0x1e0
[   20.002382] Code: 7d 78 e8 83 1b ad 00 0f 1f 44 00 00 65 48 8b 04 25 c0 7b 01 00 48 8b 80 68 08 00 00 48 8b 00 48 85 c0 0f 84 67 01 00 00 31 d2 <48> 03 50 08 48 2b 10 48 8b 40 10 48 85 c0 75 f0 48 c1 ea 0a 49 89
[   20.007834] RIP: 0010:memcpy_erms+0x6/0x10
[   20.009140] RSP: 0018:ffffb6510088bd58 EFLAGS: 00010286
[   20.010431] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   20.012338] RAX: ffa744c9c0000000 RBX: ffff964f943fc900 RCX: 0000000000000000
[   20.012341] RDX: b06967f34d3d5000 RSI: 0000000000000200 RDI: ffff964f815f55b8
[   20.012343] RBP: ffffb6510088bd78 R08: 0000000000000000 R09: fffffffffffffff0
[   20.012345] R10: 0000000000000009 R11: 0000000000000009 R12: 0000000000000009
[   20.012346] R13: ffff964f815f5540 R14: 0000000000000001 R15: 0000000000000000
[   20.012349] FS:  00007fcf9ad27640(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   20.012351] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   20.017776] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   20.020350] CR2: 00007f86e9c77914 CR3: 000000010e740003 CR4: 00000000001706e0
[   20.022454]
[   20.022455] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   20.022457] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   20.025037] Call Trace:                                                                                            [   20.025045]  do_exit+0x188/0x3d0                                                                                   [   20.027190] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80                                      [   20.029711]  do_group_exit+0x3b/0xb0                                                                               [   20.029717]  get_signal+0x16b/0x8b0
[   20.029722]  ? put_task_struct_rcu_user+0x37/0x50
[   20.032092] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   20.034167]  ? finish_task_switch.isra.0+0x1f8/0x260                                                               [   20.035719] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800
[   20.038258]  arch_do_signal_or_restart+0xd9/0x100
[   20.038728] FS:  00007f7d2233c000(0000) GS:ffff964fbbc00000(0000) knlGS:0000000000000000
[   20.041043]  ? schedule+0x4f/0xc0
[   20.041050]  exit_to_user_mode_loop+0xc4/0x160
[   20.043136] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   20.043965]  exit_to_user_mode_prepare+0x9f/0xb0
[   20.043969]  irqentry_exit_to_user_mode+0x9/0x20
[   20.043974]  irqentry_exit+0x19/0x30
[   20.044937] CR2: 0000000000002205 CR3: 0000000109950006 CR4: 00000000001706f0
[   20.047524]  exc_page_fault+0x8f/0x170
[   20.047530]  ? asm_exc_page_fault+0x8/0x30
[   20.047535]  asm_exc_page_fault+0x1e/0x30
[   20.075043] RIP: 0033:0x7fcf9db8d470
[   20.076155] Code: ff 49 39 c5 0f 84 6e 01 00 00 41 8d 47 01 89 45 88 bb ff ff ff ff 41 bc 01 00 00 00 89 5d 8c 4c 8
9 eb 0f 1f 84 00 00 00 00 00 <4c> 8b 2b 8b 0d 87 95 5e 00 44 39 f9 8b 43 10 76 0f 39 c8 73 1c 44
[   20.081766] RSP: 002b:00007fcf9ad26c80 EFLAGS: 00010283
[   20.083355] RAX: 0000000000004b79 RBX: 0000000002ba5fc0 RCX: 00007fcf9ad26c90
[   20.085512] RDX: 0000000000000000 RSI: 00000000022b1c80 RDI: 0000000000000006
[   20.087674] RBP: 00007fcf9ad26d00 R08: 000000000000006e R09: 0000000000000004
[   20.089825] R10: 00007fff6852a080 R11: 0000000000000246 R12: 0000000000000001
[   20.091978] R13: 0000000002ba5fc0 R14: 00007fff6850dad8 R15: 0000000000004b78
[   20.094135] Modules linked in: md4 cmac nls_utf8 cifs libarc4 libdes 9p fscache netfs binfmt_misc nls_iso8859_1 int
el_rapl_msr intel_rapl_common rapl input_leds serio_raw 9pnet_virtio mac_hid 9pnet efi_pstore sch_fq_codel msr drm sun
rpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async
_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni
_intel aesni_intel crypto_simd nvme cryptd virtio_net net_failover psmouse failover nvme_core
[   20.108948] ---[ end trace fd6ac84d0e480295 ]---
[   20.122433] RIP: 0010:memcpy_erms+0x6/0x10
[   20.123225] __swap_info_get: Bad swap offset entry 369044bcc3fff
[   20.123751] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1
f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   20.125537] BUG: Bad page map in process oom_reaper  pte:12df76867800000 pmd:116936067                             [   20.131200] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   20.133510] addr:0000000002a01000 vm_flags:08100073 anon_vma:ffff964f8c8cf840 mapping:0000000000000000 in[232/1637]
[   20.133514] file:(null) fault:0x0 mmap:0x0 readpage:0x0
[   20.133520] CPU: 0 PID: 29 Comm: oom_reaper Tainted: G    B D           5.13.0-27-generic #29-Ubuntu               [   20.133523] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.135129]
[   20.138103] Call Trace:
[   20.138108]  show_stack+0x52/0x58
[   20.138116]  dump_stack+0x7d/0x9c
[   20.138119]  print_bad_pte.cold+0x67/0xbf
[   20.139747] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   20.142391]  zap_pte_range+0x6c5/0x790
[   20.144611] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   20.145064]  zap_pmd_range.isra.0+0x1cc/0x2b0
[   20.145895] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80
[   20.146901]  ? release_pages+0x163/0x4b0
[   20.147972] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   20.149192]  unmap_page_range+0x218/0x3b0                                                                          [   20.151605] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800                                      [   20.152731]  __oom_reap_task_mm+0xf8/0x180                                                                         [   20.155351] FS:  00007fcf9ad27640(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000                           [   20.156644]  oom_reaper+0x274/0x350
[   20.159201] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   20.160375]  ? wait_woken+0x80/0x80
[   20.163000] CR2: 00007f86e9c77914 CR3: 000000010e740003 CR4: 00000000001706e0                                      [   20.164199]  kthread+0x11f/0x140
[   20.164205]  ? __oom_reap_task_mm+0x180/0x180
[   20.164208]  ? set_kthread_struct+0x50/0x50
[   20.164212]  ret_from_fork+0x22/0x30
[   20.183376] Fixing recursive fault but reboot is needed!
[   20.200133] general protection fault, probably for non-canonical address 0x342356e79504d0df: 0000 [#7] SMP PTI
[   20.203798] CPU: 1 PID: 361 Comm: systemd-journal Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   20.207690] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.210220] __swap_info_get: Bad swap offset entry 3690443cc3fff
[   20.212088] RIP: 0010:__kmalloc_node+0xfc/0x300
[   20.213917] BUG: Bad page map in process oom_reaper  pte:12df77867800000 pmd:116936067
[   20.217007] Code: 4c 89 f1 44 89 e6 4c 89 ff e8 70 ee ff ff 48 89 c2 80 e7 02 0f 85 76 01 00 00 48 89 55 c0 eb 3d 4
1 8b 4f 28 49 8b 3f 48 01 c1 <48> 8b 19 48 89 ce 49 33 9f b8 00 00 00 48 8d 4a 01 48 0f ce 48 31
[   20.217014] RSP: 0018:ffffb65100143858 EFLAGS: 00010202
[   20.217020] RAX: 342356e79504d07f RBX: 0000000000000000 RCX: 342356e79504d0df
[   20.219374] addr:0000000002a02000 vm_flags:08100073 anon_vma:ffff964f8c8cf840 mapping:0000000000000000 index:2a02
[   20.219378] file:(null) fault:0x0 mmap:0x0 readpage:0x0
[   20.219383] CPU: 0 PID: 29 Comm: oom_reaper Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   20.219386] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.219387] Call Trace:
[   20.219389]  show_stack+0x52/0x58
[   20.227404] RDX: 0000000000006165 RSI: 0000000000000dc0 RDI: 0000000000032100
[   20.228944]  dump_stack+0x7d/0x9c
[   20.231613] RBP: ffffb651001438a8 R08: ffff964fbbd32100 R09: 0000000000000002
[   20.234613]  print_bad_pte.cold+0x67/0xbf
[   20.236606] R10: 0000000000001460 R11: 0000000000000000 R12: 0000000000000dc0
[   20.239269]  zap_pte_range+0x6c5/0x790
[   20.241804] R13: ffff964f80042800 R14: ffffffffa650fa6d R15: ffff964f80042800
[   20.242544]  zap_pmd_range.isra.0+0x1cc/0x2b0
[   20.243828] FS:  00007fa0c4220fc0(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   20.245905]  ? release_pages+0x163/0x4b0
[   20.247196] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   20.249276]  unmap_page_range+0x218/0x3b0                                                                          [   20.250810] CR2: 00007fa0c39a3000 CR3: 0000000116d2a003 CR4: 00000000001706e0
[   20.252889]  __oom_reap_task_mm+0xf8/0x180
[   20.254327] Call Trace:
[   20.252889]  __oom_reap_task_mm+0xf8/0x180                                                               [174/1637]
[   20.254327] Call Trace:
[   20.256401]  oom_reaper+0x274/0x350                                                                                [   20.258065]  memcg_alloc_page_obj_cgroups+0x3d/0x90
[   20.260425]  ? wait_woken+0x80/0x80
[

   20.261977]  allocate_slab+0xd6/0x4a0
[   20.263650]  kthread+0x11f/0x140
[   20.265181]  ___slab_alloc+0x355/0x470
[   20.267244]  ? __oom_reap_task_mm+0x180/0x180
[   20.268785]  ? __d_alloc+0x27/0x1f0
[   20.269519]  ? set_kthread_struct+0x50/0x50
[   20.270850]  ? slab_pre_alloc_hook.constprop.0+0x71/0xe0
[   20.272294]  ret_from_fork+0x22/0x30
[   20.285751]  ? __d_alloc+0x27/0x1f0
[   20.287130]  kmem_cache_alloc+0x219/0x240
[   20.288674]  __d_alloc+0x27/0x1f0
[   20.289947]  d_alloc+0x1a/0x80                                                                                     [   20.291124]  d_alloc_parallel+0x57/0x4d0                                                                           [   20.292616]  ? allocate_slab+0x26b/0x4a0                                                                           [   20.294007]  ? __legitimize_path+0x2d/0x60                                                                         [   20.295411]  __lookup_slow+0x5b/0x150
[   20.296770]  walk_component+0x141/0x1b0
[   20.298250]  link_path_walk.part.0.constprop.0+0x23e/0x370
[   20.300337]  ? path_init+0x2c1/0x3f0                                                                               [   20.301741]  path_openat+0xb5/0x2b0
[   20.303106]  do_filp_open+0xa2/0x140
[   20.304502]  ? __check_object_size+0x1c/0x20
[   20.306144]  ? alloc_fd+0x53/0x170
[   20.307465]  do_sys_openat2+0x9b/0x150
[   20.308906]  __x64_sys_openat+0x55/0x90
[   20.310408]  do_syscall_64+0x61/0xb0
[   20.311799]  ? exit_to_user_mode_prepare+0x37/0xb0
[   20.313672]  ? syscall_exit_to_user_mode+0x27/0x50
[   20.315538]  ? do_syscall_64+0x6e/0xb0
[   20.316984]  ? do_syscall_64+0x6e/0xb0
[   20.318419]  ? do_syscall_64+0x6e/0xb0
[   20.319875]  ? syscall_exit_to_user_mode+0x27/0x50
[   20.321735]  ? exit_to_user_mode_prepare+0x37/0xb0
[   20.323573]  ? irqentry_exit_to_user_mode+0x9/0x20
[   20.325452]  ? irqentry_exit+0x19/0x30
[   20.326906]  ? exc_page_fault+0x8f/0x170
[   20.328428]  ? asm_exc_page_fault+0x8/0x30
[   20.330004]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[   20.331226] __swap_info_get: Bad swap offset entry 369043bcc3fff
[   20.331944] RIP: 0033:0x7fa0c4a6d6e4
[   20.333729] BUG: Bad page map in process oom_reaper  pte:12df78867800000 pmd:116936067
[   20.335123] Code: 24 20 eb 8f 66 90 44 89 54 24 0c e8 16 d2 f7 ff 44 8b 54 24 0c 44 89 e2 48 89 ee 41 89 c0 bf 9c f
f ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 89 44 24 0c e8 58 d2 f7 ff 8b 44
[   20.337439] addr:0000000002a03000 vm_flags:08100073 anon_vma:ffff964f8c8cf840 mapping:0000000000000000 index:2a03
[   20.344324] RSP: 002b:00007fffc5cc0f60 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[   20.344331] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa0c4a6d6e4
[   20.344333] RDX: 0000000000080802 RSI: 00005574eb13cde0 RDI: 00000000ffffff9c
[   20.344335] RBP: 00005574eb13cde0 R08: 0000000000000000 R09: ffffffffffffffff
[   20.347345] file:(null) fault:0x0 mmap:0x0 readpage:0x0
[   20.350208] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000080802
[   20.352305] CPU: 0 PID: 29 Comm: oom_reaper Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   20.354938] R13: 00000000ffffffff R14: 00005574eb13c160 R15: 00000000fffffffa                                      [   20.357020] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.359013] Modules linked in:                                                                           [116/1637]
[   20.361089] Call Trace:
[   20.361093]  show_stack+0x52/0x58                                                                                  [   20.365133]  md4
[   20.367221]  dump_stack+0x7d/0x9c
[   20.369963]  cmac
[   20.370879]  print_bad_pte.cold+0x67/0xbf
[   20.371901]  nls_utf8
[   20.372881]  zap_pte_range+0x6c5/0x790
[   20.373626]  cifs
[   20.374612]  zap_pmd_range.isra.0+0x1cc/0x2b0
[   20.375373]  libarc4
[   20.376552]  ? release_pages+0x163/0x4b0
[   20.377463]  libdes
[   20.378552]  unmap_page_range+0x218/0x3b0
[   20.379319]  9p
[   20.380608]  __oom_reap_task_mm+0xf8/0x180                                                                         [   20.381561]  fscache                                                                                               [   20.382711]  oom_reaper+0x274/0x350                                                                                [   20.383536]  netfs                                                                                                 [   20.384714]  ? wait_woken+0x80/0x80
[   20.384718]  kthread+0x11f/0x140
[   20.384723]  ? __oom_reap_task_mm+0x180/0x180
[   20.384726]  ? set_kthread_struct+0x50/0x50                                                                        [   20.384729]  ret_from_fork+0x22/0x30
[   20.403601]  binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common rapl input_leds serio_raw 9pnet_virtio mac_
hid 9pnet efi_pstore sch_fq_codel msr drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10
 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linea
r crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd nvme cryptd virtio_net net_failover psmous
e failover nvme_core
[   20.416575] ---[ end trace fd6ac84d0e480296 ]---
[   20.431398] RIP: 0010:memcpy_erms+0x6/0x10
[   20.432679] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1
f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   20.438305] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   20.438382] __swap_info_get: Bad swap offset entry 3690433cc3fff
[   20.439920] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   20.441690] BUG: Bad page map in process oom_reaper  pte:12df79867800000 pmd:116936067
[   20.443887] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   20.446186] addr:0000000002a04000 vm_flags:08100073 anon_vma:ffff964f8c8cf840 mapping:0000000000000000 index:2a04
[   20.448342] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80
[   20.451317] file:(null) fault:0x0 mmap:0x0 readpage:0x0
[   20.453477] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   20.455014] CPU: 0 PID: 29 Comm: oom_reaper Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   20.457194] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800
[   20.459836] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.459838] Call Trace:
[   20.459840]  show_stack+0x52/0x58
[   20.459845]  dump_stack+0x7d/0x9c
[   20.462010] FS:  00007fa0c4220fc0(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   20.463964]  print_bad_pte.cold+0x67/0xbf
[   20.463969]  zap_pte_range+0x6c5/0x790
[   20.463973]  zap_pmd_range.isra.0+0x1cc/0x2b0
[   20.463976]  ? release_pages+0x163/0x4b0
[   20.464736] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   20.465724]  unmap_page_range+0x218/0x3b0
[   20.466743] CR2: 00007fa0c39a3000 CR3: 0000000116d2a003 CR4: 00000000001706e0                                      [   20.469097]  __oom_reap_task_mm+0xf8/0x180
[   20.469104]  oom_reaper+0x274/0x350                                                                       [58/1637]
[   20.469107]  ? wait_woken+0x80/0x80
[   20.472311] general protection fault, probably for non-canonical address 0x342356e79504d0df: 0000 [#8] SMP PTI     [   20.472734]  kthread+0x11f/0x140
[   20.473985] CPU: 1 PID: 1 Comm: systemd Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   20.475666]  ? __oom_reap_task_mm+0x180/0x180
[   20.476919] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.478997]  ? set_kthread_struct+0x50/0x50
[   20.480261] RIP: 0010:kmem_cache_alloc_trace+0x94/0x240
[   20.481295]  ret_from_fork+0x22/0x30
[   20.482393] Code: 05 a1 03 b2 59 49 8b 00 49 83 78 10 00 48 89 45 c0 0f 84 6d 01 00 00 48 85 c0 0f 84 64 01 00 00 4
1 8b 4e 28 49 8b 3e 48 01 c1 <48> 8b 19 48 89 ce 49 33 9e b8 00 00 00 48 8d 4a 01 48 0f ce 48 31
[   20.501956] RSP: 0018:ffffb6510001fb70 EFLAGS: 00010202
[   20.503550] RAX: 342356e79504d07f RBX: ffffffffa81a2f80 RCX: 342356e79504d0df
[   20.505716] RDX: 0000000000006165 RSI: 0000000000000dc0 RDI: 0000000000032100
[   20.507896] RBP: ffffb6510001fbb8 R08: ffff964fbbd32100 R09: ffff964f88f27770
[   20.510052] R10: ffffb6510001fbf8 R11: 0000000000000005 R12: 0000000000000dc0                                      [   20.512215] R13: ffff964f80042800 R14: ffff964f80042800 R15: ffffffffa65e9b36                                      [   20.514365] FS:  00007f5db29bd600(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000                           [   20.516798] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033                                                      [   20.518532] CR2: 00007fa0c39a3000 CR3: 00000001019c4006 CR4: 00000000001706e0
[   20.520671] Call Trace:
[   20.521440]  kernfs_fop_open+0xc6/0x3a0
[   20.522620]  ? kernfs_seq_next+0x90/0x90                                                                           [   20.523831]  do_dentry_open+0x156/0x370
[   20.525011]  vfs_open+0x2d/0x30
[   20.525978]  do_open+0x205/0x3d0
[   20.526976]  ? path_init+0x2c1/0x3f0
[   20.528071]  path_openat+0x10e/0x2b0
[   20.529164]  do_filp_open+0xa2/0x140
[   20.530260]  ? __check_object_size+0x1c/0x20
[   20.531223] __swap_info_get: Bad swap offset entry 369042bcc3fff
[   20.531555]  ? alloc_fd+0x53/0x170
[   20.533333] BUG: Bad page map in process oom_reaper  pte:12df7a867800000 pmd:116936067
[   20.534384]  do_sys_openat2+0x9b/0x150
[   20.536707] addr:0000000002a05000 vm_flags:08100073 anon_vma:ffff964f8c8cf840 mapping:0000000000000000 index:2a05
[   20.537855]  __x64_sys_openat+0x55/0x90
[   20.540838] file:(null) fault:0x0 mmap:0x0 readpage:0x0
[   20.542026]  do_syscall_64+0x61/0xb0
[   20.543564] CPU: 0 PID: 29 Comm: oom_reaper Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   20.544648]  ? syscall_exit_to_user_mode+0x27/0x50
[   20.547266] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.548728]  ? __x64_sys_close+0x11/0x40
[   20.550669] Call Trace:
[   20.551882]  ? do_syscall_64+0x6e/0xb0
[   20.552618]  show_stack+0x52/0x58
[   20.553764]  ? exit_to_user_mode_prepare+0x37/0xb0
[   20.554745]  dump_stack+0x7d/0x9c
[   20.556212]  ? syscall_exit_to_user_mode+0x27/0x50
[   20.557193]  print_bad_pte.cold+0x67/0xbf
[   20.558639]  ? __x64_sys_read+0x19/0x20
[   20.559831]  zap_pte_range+0x6c5/0x790
[   20.561013]  ? do_syscall_64+0x6e/0xb0
[   20.562118]  zap_pmd_range.isra.0+0x1cc/0x2b0
[   20.563269]  ? asm_exc_page_fault+0x8/0x30
[   20.564550]  ? release_pages+0x163/0x4b0
[   20.565814]  entry_SYSCALL_64_after_hwframe+0x44/0xae                                                              [   20.566983]  unmap_page_range+0x218/0x3b0
[   20.568519] RIP: 0033:0x7f5db31486e4
[   20.569695]  __oom_reap_task_mm+0xf8/0x180
[   20.570806] Code: 24 20 eb 8f 66 90 44 89 54 24 0c e8 16 d2 f7 ff 44 8b 54 24 0c 44 89 e2 48 89 ee 41 89 c0 bf 9c ff ff ff b8 01 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 89 44 24 0c e8 58 d2 f7 ff 8b 44
[   20.572007]  oom_reaper+0x274/0x350
[   20.577599] RSP: 002b:00007ffe63a08320 EFLAGS: 00000293 ORIG_RAX: 0000000000000101
[   20.578632]  ? wait_woken+0x80/0x80
[   20.580920] RAX: ffffffffffffffda RBX: 000055e8fb729370 RCX: 00007f5db31486e4
[   20.581946]  kthread+0x11f/0x140
[   20.584122] RDX: 0000000000080000 RSI: 000055e8fb705070 RDI: 00000000ffffff9c
[   20.585082]  ? __oom_reap_task_mm+0x180/0x180
[   20.587229] RBP: 000055e8fb705070 R08: 0000000000000000 R09: 0000000000000001
[   20.588501]  ? set_kthread_struct+0x50/0x50
[   20.590641] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000080000
[   20.591873]  ret_from_fork+0x22/0x30
[   20.594031] R13: 00007f5db354ded0 R14: 000055e8fb705070 R15: 0000000000000000
[   20.597249] Modules linked in: md4 cmac nls_utf8 cifs libarc4 libdes 9p fscache netfs binfmt_misc nls_iso8859_1 intel_rapl_msr intel_rapl_common rapl input_leds serio_raw 9pnet_virtio mac_hid 9pnet efi_pstore sch_fq_codel msr drm sunrpc ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd nvme cryptd virtio_net net_failover psmouse failover nvme_core
[   20.612124] ---[ end trace fd6ac84d0e480297 ]---
[   20.628041] RIP: 0010:memcpy_erms+0x6/0x10
[   20.629328] Code: cc cc cc cc eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 fe
[   20.634980] RSP: 0018:ffffb65100803d58 EFLAGS: 00010282
[   20.636584] RAX: ffff964f96894a80 RBX: ffffb65100803e04 RCX: 00000000fff1280f
[   20.638755] RDX: 00000000fffffd8f RSI: ffff964f96982465 RDI: ffff964f96982000
[   20.640958] RBP: ffffb65100803da8 R08: ffff964f96894a80 R09: ffff964f96894a80
[   20.643131] R10: ffff964f849600c0 R11: 0000000000000010 R12: ffff964f96894a80
[   20.645325] R13: 0000000000000034 R14: ffff964f96894c40 R15: ffff964f86615800
[   20.645371] __swap_info_get: Bad swap offset entry 3690423cc3fff
[   20.647517] FS:  00007f5db29bd600(0000) GS:ffff964fbbd00000(0000) knlGS:0000000000000000
[   20.647520] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   20.649282] BUG: Bad page map in process oom_reaper  pte:12df7b867800000 pmd:116936067
[   20.651752] CR2: 00007fa0c39a3000 CR3: 00000001019c4006 CR4: 00000000001706e0
[   20.653424] addr:0000000002a06000 vm_flags:08100073 anon_vma:ffff964f8c8cf840 mapping:0000000000000000 index:2a06
[   20.660860] file:(null) fault:0x0 mmap:0x0 readpage:0x0
[   20.662388] CPU: 0 PID: 29 Comm: oom_reaper Tainted: G    B D           5.13.0-27-generic #29-Ubuntu
[   20.665039] Hardware name: OmniOS OmniOS HVM/BHYVE, BIOS 14.0 10/10/2021
[   20.666982] Call Trace:
[   20.667716]  show_stack+0x52/0x58
[   20.668705]  dump_stack+0x7d/0x9c
[   20.669686]  print_bad_pte.cold+0x67/0xbf
[   20.670867]  zap_pte_range+0x6c5/0x790
[   20.671976]  zap_pmd_range.isra.0+0x1cc/0x2b0
[   20.673261]  ? release_pages+0x163/0x4b0
[   20.674417]  unmap_page_range+0x218/0x3b0
[   20.675601]  __oom_reap_task_mm+0xf8/0x180
[   20.676810]  oom_reaper+0x274/0x350
[   20.677868]  ? wait_woken+0x80/0x80
[   20.678902]  kthread+0x11f/0x140
[   20.679231] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[   20.679868]  ? __oom_reap_task_mm+0x180/0x180
[   20.683464]  ? set_kthread_struct+0x50/0x50
[   20.684690]  ret_from_fork+0x22/0x30
[   20.685851] Kernel Offset: 0x25200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   20.703856] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---