illumos gate

How to reproduce:
1 - create VM in VMware running illumos
2 - Bring system online, and initialize server setup and delphix setup
3 - Shut down the system from the server setup menu
4 - Login to vSphere and expand one or more of the storage units.
5 - power up the device
6 - ssh to VM and notice output of zpool list -v does not reflect the the added storage unit.
7 - notice that prtvtoc on the disk shows the an increase in the number of both sectors and accessible sectors, but the number of sector count has not changed.

We have been able to reproduce this on two ESX servers. Both of the servers are running vsphere 5.5 and that seems to be the trigger to this problem.

Here's a comparison between systems running on two different ESX servers:
working correctly:
This shows that label believe the capacity is 0x1400000 blocks (10GB)
[0]> ffffff01d3946000::print struct cmlb_lun cl_map⁷
cl_map⁷ = { cl_map⁷.dkl_cylno = 0 cl_map⁷.dkl_nblk = 0x1400000 }
Comparing that to the capacity:
un_phy_blocksize = 0x200
un_blockcount = 0x1800000
The capacity here is 12GB which shows the newly added 2GB space.
Comparing this to a non-working ESX server:
This shows the label believe it has a capacity of 0x2200000 blocks (18GB)
[0]> ffffff01d6986000::print struct cmlb_lun cl_map⁷
cl_map⁷ = { cl_map⁷.dkl_cylno = 0 cl_map⁷.dkl_nblk = 0x2200000 }
The capacity of the device is:
un_phy_blocksize = 0x200
un_blockcount = 0x2200000
In this case the capacity and the label both match leaving the system thinking that LUN expansion has not taken place.

Digging further:
static int
cmlb_use_efi(struct cmlb_lun *cl, diskaddr_t capacity, int flags,
void *tg_cookie) {
<snip>
rval = DK_TG_READ(cl, buf, 1, lbasize, tg_cookie);
<snip>
nparts = ((efi_gpt_t )buf)->efi_gpt_NumberOfPartitionEntries;
gpe_lba = ((efi_gpt_t *)buf)->efi_gpt_PartitionEntryLBA;
alternate_lba = ((efi_gpt_t *)buf)->efi_gpt_AlternateLBA;
We're expecting the alternate_lba to the be the last usable LBA in the label. But instead we get:
[1]> 0xffffff01d1ca5000::print efi_gpt_t efi_gpt_AlternateLBA
efi_gpt_AlternateLBA = 0x21fffff
Later we do the following:
if (i == WD_NODE) {
/
minor number 7 corresponds to the whole disk
if the disk capacity is expanded after disk is
labeled, minor number 7 represents the capacity
indicated by the disk label.
/
cl->cl_map[i].dkl_cylno = 0;
if (alternate_lba == 1) { / * We are using backup label. Since we can * find a valid label at the end of disk, * the disk capacity is not expanded. */ cl->cl_map[i].dkl_nblk = capacity; }
else { cl->cl_map[i].dkl_nblk = alternate_lba + 1; }
cl->cl_offset[i] = 0;
}
This sets the reserved partition to the incorrect info:
[0]> ffffff01d2f45000::print struct cmlb_lun cl_map⁷
cl_map⁷ = { cl_map⁷.dkl_cylno = 0 cl_map⁷.dkl_nblk = 0x2200000 }
So it seems like it goes wrong when we read in the second block here:
rval = DK_TG_READ(cl, buf, 0, lbasize, tg_cookie);
This should give us the efi_gpt_t where we pull the alternate_lba. Somehow that data is incorrect. Time to dig into scsi read to see what is going wrong.

I'm investigating a change in ZFS that would move away from using the EFI label information to determine the amount of space that is "expandable". The idea is to determine how much additional allocatable space can be added given the new capacity of the device where allocatable size is determined by how many metaslabs we will add as a result of the expansion. This change means that we will now report the expandable space in increments of the metaslab size. For example, if a metaslab has a size of 64MB and a user expands the device by 200MB then the reported expandable space would be 192MB instead of 200MB. This more accurately reflects the space that is really being added.