illumos gate

Basically I think ZFS zpool mirroring the boot disk is not mirroring the ESP FAT partition which is sort of logical. However it's an important aspect of booting from the other disk in case of loss of one the disks in the mirror ...

Can you include a shell transcript (commands and output) of the exact steps you took to turn the single disk pool into a mirror, and the steps you used to confirm that the ESP file system was not created or populated afterwards?

I believe we have infrastructure that is supposed to install the boot loader into the ESP on mirror creation, but it's possible it didn't work for you for some reason.

Joshua M. Clulow wrote in #note-4:

Can you include a shell transcript (commands and output) of the exact steps you took to turn the single disk pool into a mirror, and the steps you used to confirm that the ESP file system was not created or populated afterwards?

I believe we have infrastructure that is supposed to install the boot loader into the ESP on mirror creation, but it's possible it didn't work for you for some reason.

The file system should be created and boot files installed, but indeed, we do not mirror ESP instances - which may be important in case you add other EFI programs/data files there. We probably should have some option to do that. Perhaps we should not have block level (volume) mirror, but file level replication (SMF service?).

Yes to actually mirror the EFI partition is a feature request, but to install the files or create a FAT filesystem looks more like a bug to me so I opened a separate issue, or in fact a few different bug reports. the OpenIndiana installer does not seem to currently installation onto a mirrored pair of disks, although that is not an Illumos issue but an OI installer issue I think

Checking an old logfile, I think I added the second disk without formatting it. I received a disk, put it in the system, and then format reported the following :

For the disk onto which rpool was already installed (installed by the OI text installer to a single SATA target disk ):

Specify disk (enter its number): 0
selecting c5t0d0
[disk formatted]
/dev/dsk/c5t0d0s1 is part of active ZFS pool rpool. Please see zpool(8).

For the second disk, I had :

Specify disk (enter its number)[0]: 1
selecting c5t2d0
[disk formatted]
No Solaris fdisk partition found.

This seemed logical as the disk was new, and had nothing on it.

As far as I understood, if I use "whole disk mode" in the zpool attach command, I am not supposed to label or format the disk myself, so I didn't do that.

Also the name of the second disk had some info in it like : < blahblah cyl 60799 alt 2 hd 255 sec 63>

I immediately ran :

# zpool attach rpool c5t0d0 c5t2d0

So I used "whole disk" mode to attach the empty second disk (which I did not format) immediately to the rpool.

That seemed to work fine :

  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.

After a while I had:

config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0
            c5t2d0  ONLINE       0     0     0

errors: No known data errors

After the attach and before rebooting, I noticed in format that the name of the second disk had changed

1. c5t2d0 <blahblah-465.76GB>
Specify disk (enter its number): 1
selecting c5t2d0
[disk formatted]
/dev/dsk/c5t2d0s1 is part of active ZFS pool rpool. Please see zpool(8).

So format correctly reported that I had a second 500GB disk with "disk formatted" which the ZFS attach somehow did itself.

I didn't ran "format verify" at the time but I ran "format partition print" and had:

partition> print
Current partition table (original):
Total disk sectors available: 976756717 + 16384 (reserved sectors)

Part      Tag    Flag     First Sector         Size         Last Sector
  0     system    wm               256      256.00MB          524543    
  1        usr    wm            524544      465.50GB          976756750    
  2 unassigned    wm                 0           0               0    
  3 unassigned    wm                 0           0               0    
  4 unassigned    wm                 0           0               0    
  5 unassigned    wm                 0           0               0    
  6 unassigned    wm                 0           0               0    
  8   reserved    wm         976756751        8.00MB          976773134   

Isn't the partition 0 in the above the 256MB EFI ESP ?

Anyway, I rebooted my system but the Dell UEFI boot menu did NOT find anything except for the first disk, which booted fine into the working ZFS mirror.

However because I thought that OpenIndiana / Illumos did not mirror my ESP/EFI, I manually ran the following commands :

mount -F pcfs /dev/dsk/c5t0d0s0 /tmp/mnt
mount -F pcfs /dev/dsk/c5t2d0s0 /mnt
# cd /tmp/mnt
# find . | cpio -mdvp /mnt
cpio: Cannot chown() "/mnt/EFI", errno 22, Invalid argument
/mnt/EFI
cpio: Cannot chown() "/mnt/EFI/Boot", errno 22, Invalid argument
/mnt/EFI/Boot
cpio: Cannot chown() "/mnt/EFI/Boot/bootx64.efi", errno 22, Invalid argument
/mnt/EFI/Boot/bootx64.efi
cpio: Cannot chown() "/mnt/EFI/Boot/bootia32.efi", errno 22, Invalid argument
/mnt/EFI/Boot/bootia32.efi
35360 blocks
4 error(s)
# find /mnt
/mnt/EFI
/mnt/EFI/Boot
/mnt/EFI/Boot/bootx64.efi
/mnt/EFI/Boot/bootia32.efi
# umount /mnt

Presumably this manual cpio copy is not the right way to do it, but it seemed to work in the Dell UEFI because the Dell BIOS then recognized that there was an EFI / ESP on the second disk on the next reboot and it showed up in the Dell boot menu.

Anyway what I can do is reinstall everything and try some of the things that I reported again with a more recent OpenIndiana version but this will take time.

Also I think a ran mkfs -F pcfs /dev/rdsk/c5t2d0s0 myself as far as I remember after the zpool attach and reboot.

My disk controller number has meanwhile changed because I think I did multiple new reinstalls, but the partitioning currently is :

# gdisk -l /dev/dsk/c4t2d0
Number  Start (sector)    End (sector)  Size       Code  Name
   1             256          524543   256.0 MiB   EF00  loader
   2          524544       976756750   465.5 GiB   BF01  zfs
   9       976756751       976773134   8.0 MiB     BF07