Project

General

Profile

Actions

Bug #14816

closed

Boot from NVMe scans all devices after 14688

Added by Andy Fiddaman 2 months ago. Updated about 1 month ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
driver - device drivers
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

Whilst working on syncing the latest changes from gate into OmniOS, I spotted this new behaviour when booting a VM from the new pieces:

OmniOS r151043 Version omnios-upstream_merge-2022071401-e9d814231b7 64-bit
Copyright (c) 2012-2017 OmniTI Computer Consulting, Inc.
Copyright (c) 2017-2022 OmniOS Community Edition (OmniOSce) Association.
DEBUG enabled
Warning: /kernel/drv/amd64/adpu320 uses deprecated _depends_on interface.
Please notify module developer or vendor.
Warning: /kernel/drv/amd64/glm uses deprecated _depends_on interface.
Please notify module developer or vendor.
Warning: /kernel/drv/amd64/lsimega uses deprecated _depends_on interface.
Please notify module developer or vendor.
Hostname: bloody

The warnings from these closed-source drivers are known, but why are those drivers being loaded during boot?

I traced the driver load using a kmdb breakpoint to dump the stack and module name whenever mod_load() was called:
(this was happening before the dtrace driver loaded so, although I tried anonymous trace first, it wasn't much help)

Here's the first of those closed source drivers being loaded:

[0]> mod_load::bp -c '<rdi::print struct modctl mod_modname; ::stack; :c'
...
mod_modname = 0xfffffe2d540c6da0 "adpu320" 
mod_load()
modrload+0xfa(fffffffffbf16a4e, fffffe2d4a078900, 0)
modload+0x17(fffffffffbf16a4e, fffffe2d4a078900)
mod_hold_dev_by_major+0x9f(6)
ddi_hold_driver+0x11(6)
ddi_hold_installed_driver+0x28(6)
attach_driver_by_parent+0x4d(bf, fffffe2d4f2db340)
ddi_hold_installed_driver+0xee(bf)
e_ddi_devid_hold_by_major+0x25(bf)
e_ddi_devid_hold_installed_driver+0xab(fffffe2d53b37b50)
e_ddi_devid_discovery+0x192(fffffe2d53b37b50)
ddi_lyr_devid_to_devlist+0x98(fffffe2d53b37b50, fffffe2d540c6ac0, fffffffffbcb2d78, fffffffffbcb2d80)
ldi_vp_from_devid+0x61(fffffe2d53b37b50, fffffe2d540c6ac0, fffffffffbcb2e00)
ldi_open_by_devid+0x81(fffffe2d53b37b50, fffffe2d540c6ac0, 1, fffffe2d53d02dd8, fffffffffbcb2e98, fffffe2d557209c0)
zfs`vdev_disk_read_rootlabel+0x1cb(fffffffffbc02010, fffffe2d54160140, fffffffffbcb2f38)
zfs`spa_generate_rootconf+0x2c(fffffffffbc02010, fffffe2d54160140, fffffffffbcb2fc0, 3c53f0565dc57b4b)
zfs`spa_import_rootpool+0x3f(fffffffffbc02010, fffffe2d54160140, 3c53f0565dc57b4b, 8f6a7677cf85ca97)
zfs`zfs_mountroot+0xd7(fffffffffbfd0cc0, 0)

The driver is being loaded because ldi_open_by_devid() is falling back onto device discovery rather than being able to use the devid cache.
I tried moving aside the contents of /etc/devices/, but no change in behaviour.

To rule out something peculiar with my test VM, another OmniOS developer built the working merge branch on a physical machine with NVMe boot, and confirmed the same symptom present there.

Next, adding these lines to a new file at /etc/system.d/devid

set devid_log_finds = 1
set devid_log_paths = 1
set devid_log_matches = 1
set devid_log_failures = 1
set devid_log_lookups = 1
set devid_log_stale = 1

yielded these boot messages:

OmniOS r151043 Version omnios-upstream_merge-2022071401-e9d814231b7 64-bit
Copyright (c) 2012-2017 OmniTI Computer Consulting, Inc.
Copyright (c) 2017-2022 OmniOS Community Edition (OmniOSce) Association.
DEBUG enabled
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
lookup /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
stale device reference: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
no devid found: id1,kdev@w589cfc20095f0001
Warning: /kernel/drv/amd64/adpu320 uses deprecated _depends_on interface.
Please notify module developer or vendor.
Warning: /kernel/drv/amd64/glm uses deprecated _depends_on interface.
Please notify module developer or vendor.
Warning: /kernel/drv/amd64/lsimega uses deprecated _depends_on interface.
Please notify module developer or vendor.
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
Hostname: bloody

bloody console login:

of which the important line is:

stale device reference: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001

although this appears correct:

bloody# zdb -C rpool | egrep 'path|devid'
                path: '/dev/dsk/c6t589CFC20095F0001d0s1'
                devid: 'id1,kdev@w589cfc20095f0001/b'
                phys_path: '/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0:b'
bloody# zpool status rpool
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:43:48 with 0 errors on Wed Nov 17 20:10:03 2021
config:

        NAME                     STATE     READ WRITE CKSUM
        rpool                    ONLINE       0     0     0
          c6t589CFC20095F0001d0  ONLINE       0     0     0

errors: No known data errors
bloody# ls -l /dev/dsk/c6t589CFC20095F0001d0s1
lrwxrwxrwx   1 root     root          64 Jul  6 13:50 /dev/dsk/c6t589CFC20095F0001d0s1 -> ../../devices/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0:b

I reverted #14688 and re-tested, and the problem no longer shows up:

OmniOS r151043 Version omnios-upstream_merge-2022071401-e9d814231b7 64-bit
Copyright (c) 2012-2017 OmniTI Computer Consulting, Inc.
Copyright (c) 2017-2022 OmniOS Community Edition (OmniOSce) Association.
DEBUG enabled
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
lookup /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
find: id1,kdev@w589cfc20095f0001
find: /pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0 id1,kdev@w589cfc20095f0001
/pci@0,0/pcifb5d,a0a@4/blkdev@w589CFC20095F0001,0
Hostname: bloody

bloody console login:

Related issues

Related to illumos gate - Bug #14688: nvme blkdev attach/detach trips assertion in ndi_devi_unconfig_one()ClosedHans Rosenfeld

Actions
Actions #1

Updated by Andy Fiddaman 2 months ago

  • Related to Bug #14688: nvme blkdev attach/detach trips assertion in ndi_devi_unconfig_one() added
Actions #2

Updated by Andy Fiddaman 2 months ago

  • Description updated (diff)
Actions #3

Updated by Andy Fiddaman 2 months ago

  • Description updated (diff)
Actions #4

Updated by Electric Monk 2 months ago

  • Gerrit CR set to 2244
Actions #5

Updated by Andy Fiddaman 2 months ago

  • Description updated (diff)
Actions #6

Updated by Andy Fiddaman 2 months ago

#14688 was fixing a panic in a case of an NVMe device with multiple namespaces and where the first one attached but a subsequent one failed to attach. The fix was to perform the namespace attachments asynchronously (via a taskq), after returning success from the driver's attach method. The problem here is that none of the namespaces are attached when this method returns and so the device ID that is being looked for is not yet present.

I've reworked #14688 to remove the asynchronous part and so that:

  • If the first namespace fails to attach, the driver fails to attach (this puts us back to the original behaviour before #14688);
  • If the first namespace attaches, then any other namespaces which fail to attach are basically skipped and generate a warning message, but the driver attach succeeds. As per the analysis in #14688, it isn't possible to unwind properly in this case.

I've tested this in the nvme environments I have, all of which are just a single namespace, and a full device scan is no longer performed.

Actions #7

Updated by Hans Rosenfeld about 2 months ago

Testing: I ported this to Tintri's code base. I reverted any other NVMe changes recently made to address various aspects of this issue, verifying the system trips the assertion (#14688). Then I added this change on top, the system no longer tripped the assertion and instead handled attach failure of namespaces gracefully.

Actions #8

Updated by Electric Monk about 1 month ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 30a4bccb45421aa759c9731e9094f2fa13f321c7

commit  30a4bccb45421aa759c9731e9094f2fa13f321c7
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2022-08-12T08:56:37.000Z

    14816 Boot from NVMe scans all devices after 14688
    Reviewed by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
    Reviewed by: Garrett D'Amore <garrett@damore.org>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF