illumos gate

The arc caches data based on the spa guid, and "aux" devices (spare & l2arc) use the guid to determine pool membership. There is also some interaction with FMA. This makes re-guid-ing a live pool somewhat problematic, so my preference would be for an import option instead. e.g. "zpool import -g tank".

We are well aware of the use of GUID in the ARC. To fix this, George and I came up with a new ephemeral GUID that only lives in kernel memory, but is created for the purposes of the ARC at import time. There are changes to the ARC for this. I have diffs for this already, and it is tested and working well. The first version of the webrev is up at http://dev1.illumos.org/~gdamore/reguid/ - there are some updates to this that I will post shortly (thanks to updates from George), but fundamentally the reguid logic including the updates to the ARC code are there. No problems at all. :-)

For the record, we started thinking of the reguid at import time, but the problem with this approach really was how to handle the situation where one or more vdevs are unavailable. We can import a pool that is missing vdevs just fine (e.g. half of a mirror), but until we find ourselves in a chicken and egg situation dealing with those offline vdevs at the time we would want to reguid. Its also simpler to do the reguid after the pool is imported, as then we can make use of "atomicity" of ZFS updates to change the GUID, whereas to do it at import time we would have to add a lot of logic to do this properly before we had the pool imported.

So, while its counter-intuitive to reguid the online pool, it turns out to be far far less problematic. The ARC concern you raised turns out to be fairly trivial to address as you can see from my diffs.

Oh, wrt aux devices and guid use, George and I analyzed those code paths -- and it appears that they make use of functions to inquire guid under control of the appropriate locks -- so when we change the guid those code paths behave properly and see the guid. In particular, it doesn't appear that they "cache" the guid locally, unlike the ARC. So we're pretty sure we got this all right. :-)

Finally, wrt FMA... while we don't know what Oracle's code does, the current FMA code appears to be unaffected by these changes as well. A conversation with Eric about this and our own grok'ing of the code seemed to agree on this point. In particular, it makes use of spa_guid() (just as the aux device code does), so it will see the new guid. If a fault ensues while a guid change is in flight, there is a race there, but that's to be expected. (We expected pool reguids to be very very rare and only done with administrative action.)

There is one other tidbit here, which is for correlation of guids outside of the kernel, we have added a sysevent so that observers can notice the guid change. Hopefully nobody is depending too much on the guid though.

Excellent, thanks for the detailed explanation Garrett. Glad it works in online mode. I think it can work for the import case as well: in spa_load_impl() wait until the pool is loaded and only reguid if all devices are healthy.

True, but in order to load the pool, you need to have non-conflicting GUIDs, which would lead to a chicken-and-egg problem for some of the use cases we wanted for this for.