illumos gate

Which filesystems did you try? In my experience both UFS and ZFS require the -O option to mount over the top of an another mount point (unless it's a remount) or a non empty directory.

I can point out where this is enforced in the code:

UFS: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/ufs/ufs_vfsops.c#L282
ZFS: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/zfs_vfsops.c#L1598

Both zfs(1m) and mount(1m) should be failing to mount over a non-empty directory. Maybe mount(1m) is doing something different like passing -o remount?

Andrew Stormont wrote:

Which filesystems did you try?

Several. The example with ZFS is in the Description. An example with UFS is here:

# mount | grep ^/mnt
# mkfile 100m /var/tmp/fs1
# lofiadm -a /var/tmp/fs1
/dev/lofi/1
# newfs /dev/lofi/1
newfs: construct a new file system /dev/rlofi/1: (y/n)? y
/dev/rlofi/1:   204600 sectors in 341 cylinders of 1 tracks, 600 sectors
        99,9MB in 22 cyl groups (16 c/g, 4,69MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,
 115232, 124832, 134432, 144032, 153632, 163232, 172832, 182432, 192032, 201632
# mount /dev/lofi/1 /mnt
# mkdir /mnt/a
# touch /mnt/a/b
# mkfile 100m /var/tmp/fs2
# lofiadm -a /var/tmp/fs2
/dev/lofi/2
# newfs /dev/lofi/2
newfs: construct a new file system /dev/rlofi/2: (y/n)? y
/dev/rlofi/2:   204600 sectors in 341 cylinders of 1 tracks, 600 sectors
        99,9MB in 22 cyl groups (16 c/g, 4,69MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,
 115232, 124832, 134432, 144032, 153632, 163232, 172832, 182432, 192032, 201632
# mount /dev/lofi/2 /mnt/a
# ls -la /mnt/a
total 10
drwxr-xr-x 3 root root  512 mar  9 13:27 .
drwxr-xr-x 4 root root  512 mar  9 13:26 ..
drwx------ 2 root root 8192 mar  9 13:27 lost+found
# umount /mnt/a
# ls -la /mnt/a
total 2
drwxr-xr-x 2 root root 512 mar  9 13:26 .
drwxr-xr-x 4 root root 512 mar  9 13:26 ..
-rw-r--r-- 1 root root   0 mar  9 13:26 b
#

Because of this bug, here is a simple way how to make a system unbootable:

# zfs create -p rpool/test/a/b
# zfs umount rpool/test
# zfs mount rpool/test/a/b
# zfs mount -a
cannot mount '/rpool/test': directory is not empty
cannot mount '/rpool/test/a': directory is not empty
#

After the reboot the filesystem/local SMF service will go to maintenance and because of that the sshd won't start (and most of the services as well), so you just killed your remote server.

I think the zfs command is doing the right thing here. The boot script should not be erroring out though so quickly. I have seen this myself when importing another rpool with -N option (which is not persistent - thought I'd filed a bug for it but can't find anything) and rebooting with it still imported.

I think these are all separate but related bugs.

The "zfs mount" is failing on non-empty mount points because of this piece of code in zfs_mount() (in libzfs):

294    /*
295     * Determine if the mountpoint is empty.  If so, refuse to perform the
296     * mount.  We don't perform this check if MS_OVERLAY is specified, which
297     * would defeat the point.  We also avoid this check if 'remount' is
298     * specified.
299     */
300    if ((flags & MS_OVERLAY) == 0 &&
301        strstr(mntopts, MNTOPT_REMOUNT) == NULL &&
302        !dir_is_empty(mountpoint)) {
303        zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
304            "directory is not empty"));
305        return (zfs_error_fmt(hdl, EZFS_MOUNTFAILED,
306            dgettext(TEXT_DOMAIN, "cannot mount '%s'"), mountpoint));
307    }

That seems perfectly reasonable to me. Here is the matching code from the zfs_mount function in zfs_vfsops.c, lines 1597-1604

    mutex_enter(&mvp->v_lock);
    if ((uap->flags & MS_REMOUNT) == 0 &&
        (uap->flags & MS_OVERLAY) == 0 &&
        (mvp->v_count != 1 || (mvp->v_flag & VROOT))) {
        mutex_exit(&mvp->v_lock);
        return (SET_ERROR(EBUSY));
    }
    mutex_exit(&mvp->v_lock);

Most of the other file systems have something similar in their mount functions too.

EDIT: I see that I have mentioned this before. The zfs command is working correctly.

Andrew Stormont wrote:

That seems perfectly reasonable to me. Here is the matching code from the zfs_mount function in zfs_vfsops.c, lines 1597-1604

[...]

Most of the other file systems have something similar in their mount functions too.

Note: The matching code in zfs_vfsops.c allows to mount in a case the mount point is not empty. There is no problem in zfs_vfsops.c.

The zfs(1m) man page does not suggest that the mount point directory must be empty. The man page says this about the mount subcommand:

       zfs mount [-vO] [-o options] -a | filesystem
           Mounts ZFS file systems. Invoked automatically as part of the boot
           process.

           -o options
               An optional, comma-separated list of mount options to use
               temporarily for the duration of the mount. See the "Temporary
               Mount Point Properties" section for details.

           -O
               Perform an overlay mount. See mount(1M) for more information.

           -v
               Report mount progress.

           -a
               Mount all available ZFS file systems. Invoked automatically as
               part of the boot process.

           filesystem
               Mount the specified filesystem.