illumos gate

One of my backup machines is now in a panic-reboot cycle with this error.
joyent_20200813T030805Z
Supermicro Super Server H11SSL-C
AMD EPYC 7401P 24-Core Processor CPU
LSI SAS3008 PCI-Express Fusion-MPT SAS-3
Chelsio Communications Inc T580-SO-CR Unified Wire Ethernet Controller
Sanima/Newisys/Viking Storage 106 disk JBOD

In single user mode ZFS checks out, passes scrubs etc. In multi-user mode it reports I/O errors, pool corruption, bad disks, and other nasties.

3 Other systems with the same config do no exhibit these errors yet. One other system is connected to the same JBOD but different zpools.

--- panic message ---
backup004 ttya login:
panic[cpu5]/thread=fffffe8579a29400: zfs: accessing past end of object 305/59ca (size=382976 access=382955+29)

fffffe00bd33e900 genunix:vcmn_err+42 ()
fffffe00bd33e970 zfs:zfs_panic_recover+65 ()
fffffe00bd33ea20 zfs:dmu_buf_hold_array_by_dnode+2b2 ()
fffffe00bd33eac0 zfs:dmu_write_uio_dnode+52 ()
fffffe00bd33eb10 zfs:dmu_write_uio_dbuf+5d ()
fffffe00bd33ed10 zfs:zfs_write+9ba ()
fffffe00bd33edb0 genunix:fop_write+111 ()
fffffe00bd33ee80 genunix:write+2c6 ()
fffffe00bd33eeb0 genunix:write32+15 ()
fffffe00bd33ef10 unix:brand_sys_sysenter+1dc ()

--- cut here ---

--- core status ---
[root@00-07-43-52-c1-90 /var/crash/volatile]# mdb -k 28
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc pcplusmp scsi_vhci ufs ip hook neti sockfs arp usba uhci smbios mm sd fctl stmf_sbd stmf zfs lofs idm mpt_sas sata crypto random cpc fcp logindmux ptm kvm sppp nsmb smbsrv nfs ipc ]

::status
debugging crash dump vmcore.28 (64-bit) from backup004
operating system: 5.11 joyent_20200813T030805Z (i86pc)
git branch: release-20200813
git rev: 09f210323354e07890b312da9b4f5e33ae6b6df5
image uuid: (not set)
panic message: zfs: accessing past end of object 684/59ca (size=518656 access=518656+5)
dump content: kernel pages only

::system
set ibft_noprobe=0x1 [0t1]
set noexec_user_stack=0x1 [0t1]
set noexec_user_stack_log=0x1 [0t1]
set rlim_fd_cur=0x10000 [0t65536]
set idle_cpu_no_deep_c=0x1 [0t1]
set ip:ip_squeue_fanout=0x1 [0t1]
set pcplusmp:apic_panic_on_nmi=0x1 [0t1]
set apix:apic_panic_on_nmi=0x1 [0t1]
set dump_plat_mincpu=0x0 [0t0]
set dump_bzip2_level=0x1 [0t1]
set pcplusmp:apic_timer_preferred_mode=0x0 [0t0]
set apix:apic_timer_preferred_mode=0x0 [0t0]
set dump_metrics_on=0x1 [0t1]
set sata:sata_auto_online=0x1 [0t1]
set sd:sd_io_time=0xa [0t10]
set hires_tick=0x1 [0t1]
set zfs:zfs_multihost_interval=0xffff [0t65535]
set zfs:zfs_resilver_delay=0x0 [0t0]
set zfs:zfs_resilver_min_time_ms=0x0 [0t0]
set zfs:zfs_scrub_delay=0x0 [0t0]
set zfs:zfs_arc_max=0xc80000000 [0t53687091200]
set ddi_msix_alloc_limit=0x8 [0t8]
set pcplusmp:apic_intr_policy=0x1 [0t1]
set ip_squeue_soft_ring=0x1 [0t1]

--- cut here ---