To add some more concrete detail, if we call ndi_devi_config() before vfs_mountroot(), we see the following effects:

root@oi0:~# svcs -xv
svc:/network/iscsi/initiator:default (iSCSI initiator daemon)
 State: maintenance since October 20, 2019 at 12:00:23 PM PDT
Reason: Start method failed repeatedly, last exited with status 1.
   See: http://illumos.org/msg/SMF-8000-KS
   See: man -M /usr/share/man -s 7D iscsi
   See: /var/svc/log/network-iscsi-initiator:default.log
Impact: This service is not running.

Looking at the log:

[ Oct 20 12:00:23 Executing start method ("/lib/svc/method/iscsi-initiator start"). ]
[ Oct 20 12:00:23 Method "start" exited with status 1. ]
[ Oct 20 12:00:23 Executing start method ("/lib/svc/method/iscsi-initiator start"). ]
[ Oct 20 12:00:23 Method "start" exited with status 1. ]
[ Oct 20 12:00:23 Executing start method ("/lib/svc/method/iscsi-initiator start"). ]
[ Oct 20 12:00:23 Method "start" exited with status 1. ]

Trying to start the daemon manually:

root@oi0:~# truss -t open,ioctl -f /lib/svc/method/iscsid
1365:   open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
1365:   open("/lib/libc.so.1", O_RDONLY)                = 3
1365:   open("/usr/lib/locale//en_US.UTF-8/LC_CTYPE/LCL_DATA", O_RDONLY) = 3
1365:   open("/usr/lib/locale//en_US.UTF-8/LC_NUMERIC/LCL_DATA", O_RDONLY) = 3
1365:   open("/usr/lib/locale//en_US.UTF-8/LC_TIME/LCL_DATA", O_RDONLY) = 3
1365:   open("/usr/lib/locale//en_US.UTF-8/LC_COLLATE/LCL_DATA", O_RDONLY) = 3
1365:   open("/usr/lib/locale//en_US.UTF-8/LC_MONETARY/LCL_DATA", O_RDONLY) = 3
1365:   open("/usr/lib/locale//en_US.UTF-8/LC_MESSAGES/LCL_DATA", O_RDONLY) = 3
1366:   open("/dev/null", O_RDWR)                       = 4
1366/1:         open("/devices/iscsi:devctl", O_RDWR)           = 6
1366/1:         ioctl(6, (('i'<<24)|('S'<<16)|('C'<<8)|34), 0x080632E0) Err#14 EFAULT
1366/1:         open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/messages.mo", O_RDONLY) Err#2 ENOENT
1366/1:         open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/SUNW_OST_OSLIB.mo", O_RDONLY) Err#2 ENOENT
1365/2:             Received signal #18, SIGCLD, in lwp_park() [caught]
1365/2:               siginfo: SIGCLD CLD_EXITED pid=1366 status=0xFFFFFFFF

ioctl 34 is:

#define ISCSI_SMF_ONLINE                (ISCSI_IOCTL | 34)

In iscsi_ioctl(), a suspicious and ultimately incorrect comment:

        case ISCSI_SMF_ONLINE:
...
                /* just a theoretical case */
                if (ihp->hba_persistent_loaded == B_FALSE) {
                        rtn = EFAULT;
                        break;
                }

Indeed, we're stuck attached but without the persistent store loaded:

> *iscsi_state::walk softstate | ::print iscsi_hba_t hba_persistent_loaded
hba_persistent_loaded = 0 (0)

By forcing the module to detach, we can clear the service:

# modinfo | grep iscsi
 49 fffffffff7838000  349f8 108   1  iscsi (iSCSI Initiator v-1.55)
# modunload -i 49 
# svcadm clear iscsi/initiator
# svcs iscsi/initiator
STATE          STIME    FMRI
online         12:21:52 svc:/network/iscsi/initiator:default

Testing Notes¶

In order to test this change, I first replicated the symptoms using an unconditional pre-mountroot call to code which attempts to attach all driver instances:

@@ -504,6 +511,18 @@ main(void)
        process_cache = kmem_cache_create("process_cache", sizeof (proc_t),
            0, NULL, NULL, NULL, NULL, NULL, 0);

+       /*
+        * XXX Force for iSCSI testing...
+        */
+       extern void preroot_walk_block_devices(
+           int (*f)(const char *, void *), void *arg);
+       printf("FORCE preroot_walk_block_devices ...\n");
+       preroot_walk_block_devices(xxx_preroot_walker, NULL);
+       printf("... END FORCED preroot_walk_block_devices\n");
+       /*
+        * XXX
+        */
+
        vfs_mountroot();        /* Mount the root file system */
        errorq_init();          /* after vfs_mountroot() so DDI root is ready */
        cpu_kstat_init(CPU);    /* after vfs_mountroot() so TOD is valid */

The salient part of preroot_walk_block_devices() is:

+       (void) ndi_devi_config(ddi_root_node(), NDI_CONFIG | NDI_DEVI_PERSIST |
+           NDI_NO_EVENT | NDI_DRV_CONF_REPROBE);

The sequence of builds and tests I performed were as follows:

• configure a stock OpenIndiana system as an iSCSI initiator and reboot
  • everything is working as expected (iSCSI service online, volumes visible and functional)
• build with only the addition of the preroot_walk_block_devices() call and update the system
  • next boot, the service is offline as described in the comments
• build with the addition of the change
  • next boot, the service is online and the volumes are visible and functional once more
• final RTI build with just the change at issue
  • booted and checked that service was online and volumes were visible