Bug #5579
zfs mount is inconsistent with mount(1m)
0%
Description
The zfs mount command behaves differently than other mount operations. All filesystems, when mounted using the mount(1m) command, have no problem to mount even in a case the mountpoint directory is not empty. This is also documented in the mount(1m) man page:
mount attaches a file system to the file system hierarchy at the mount_point, which is the pathname of a directory. If mount_point has any contents prior to the mount operation, these are hidden until the file system is unmounted.
Similar sentence is in mount man pages for particular filesystems, for example this is in mount_hsfs(1m):
mount attaches an ISO 9660 filesystem (the High Sierra file system, hsfs, is a draft predecessor to ISO 9660, so the name reflects the filesystem's history) to the file system hierarchy at the mount_point, which is the pathname of a directory. If mount_point has any contents prior to the mount operation, these are hidden until the file system is unmounted.
Actually, the zfs filesystem works similarly when mounted using the mount(1m) command, for example:
# ls -al 1 total 2 drwxr-xr-x 2 root root 3 jan 20 16:18 . drwxr-xr-x 4 root root 4 jan 20 20:39 .. -rw-r--r-- 1 root root 0 jan 20 16:18 a # mount -F zfs rpool/export/test/1 1 # ls -al 1 total 1 drwxr-xr-x 2 root root 2 jan 20 16:02 . drwxr-xr-x 4 root root 4 jan 20 20:39 .. # umount 1 #
The only difference is when the zfs mount command is used:
# ls -al 1 total 2 drwxr-xr-x 2 root root 3 jan 20 16:18 . drwxr-xr-x 4 root root 4 jan 20 20:39 .. -rw-r--r-- 1 root root 0 jan 20 16:18 a # zfs mount rpool/export/test/1 cannot mount '/export/test/1': directory is not empty # zfs mount -O rpool/export/test/1 # ls -al 1 total 1 drwxr-xr-x 2 root root 2 jan 20 16:02 . drwxr-xr-x 4 root root 4 jan 20 20:39 .. #
As we see from the test, I had to use -O (overlay mount) to achieve the desired behavior.
I see few issues with the current zfs mount behavior:
- The -O flag is required in a case the mount directory is not empty. This is different behavior when compared to the mount(1m) command.
- When the -O flag is not used, the zfs(1m) produces error "directory is not empty", but according to the mount(1m) man page the error when the -O flag is not used (and is needed) should be "device busy".
- The zfs mount requires the -O flag for a case when we do not do the real overlay mount. This is confusing.
I'd expect that both zfs mount and mount(1m) commands behaves similarly, IOW, the zfs mount (without the -O flag) should not fail in our test case above.
The described difference between zfs mount and mount(1m) could cause some zfs filesystems are not mounted on boot.
Related issues
Updated by Andrew Stormont almost 6 years ago
Which filesystems did you try? In my experience both UFS and ZFS require the -O option to mount over the top of an another mount point (unless it's a remount) or a non empty directory.
I can point out where this is enforced in the code:
UFS: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/ufs/ufs_vfsops.c#L282
ZFS: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/zfs_vfsops.c#L1598
Both zfs(1m) and mount(1m) should be failing to mount over a non-empty directory. Maybe mount(1m) is doing something different like passing -o remount?
Updated by Marcel Telka almost 6 years ago
Andrew Stormont wrote:
Which filesystems did you try?
Several. The example with ZFS is in the Description. An example with UFS is here:
# mount | grep ^/mnt # mkfile 100m /var/tmp/fs1 # lofiadm -a /var/tmp/fs1 /dev/lofi/1 # newfs /dev/lofi/1 newfs: construct a new file system /dev/rlofi/1: (y/n)? y /dev/rlofi/1: 204600 sectors in 341 cylinders of 1 tracks, 600 sectors 99,9MB in 22 cyl groups (16 c/g, 4,69MB/g, 2240 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432, 115232, 124832, 134432, 144032, 153632, 163232, 172832, 182432, 192032, 201632 # mount /dev/lofi/1 /mnt # mkdir /mnt/a # touch /mnt/a/b # mkfile 100m /var/tmp/fs2 # lofiadm -a /var/tmp/fs2 /dev/lofi/2 # newfs /dev/lofi/2 newfs: construct a new file system /dev/rlofi/2: (y/n)? y /dev/rlofi/2: 204600 sectors in 341 cylinders of 1 tracks, 600 sectors 99,9MB in 22 cyl groups (16 c/g, 4,69MB/g, 2240 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432, 115232, 124832, 134432, 144032, 153632, 163232, 172832, 182432, 192032, 201632 # mount /dev/lofi/2 /mnt/a # ls -la /mnt/a total 10 drwxr-xr-x 3 root root 512 mar 9 13:27 . drwxr-xr-x 4 root root 512 mar 9 13:26 .. drwx------ 2 root root 8192 mar 9 13:27 lost+found # umount /mnt/a # ls -la /mnt/a total 2 drwxr-xr-x 2 root root 512 mar 9 13:26 . drwxr-xr-x 4 root root 512 mar 9 13:26 .. -rw-r--r-- 1 root root 0 mar 9 13:26 b #
Updated by Marcel Telka almost 6 years ago
Because of this bug, here is a simple way how to make a system unbootable:
# zfs create -p rpool/test/a/b # zfs umount rpool/test # zfs mount rpool/test/a/b # zfs mount -a cannot mount '/rpool/test': directory is not empty cannot mount '/rpool/test/a': directory is not empty #
After the reboot the filesystem/local SMF service will go to maintenance and because of that the sshd won't start (and most of the services as well), so you just killed your remote server.
Updated by Andrew Stormont almost 6 years ago
I think the zfs command is doing the right thing here. The boot script should not be erroring out though so quickly. I have seen this myself when importing another rpool with -N option (which is not persistent - thought I'd filed a bug for it but can't find anything) and rebooting with it still imported.
I think these are all separate but related bugs.
Updated by Marcel Telka almost 6 years ago
The "zfs mount" is failing on non-empty mount points because of this piece of code in zfs_mount() (in libzfs):
294 /* 295 * Determine if the mountpoint is empty. If so, refuse to perform the 296 * mount. We don't perform this check if MS_OVERLAY is specified, which 297 * would defeat the point. We also avoid this check if 'remount' is 298 * specified. 299 */ 300 if ((flags & MS_OVERLAY) == 0 && 301 strstr(mntopts, MNTOPT_REMOUNT) == NULL && 302 !dir_is_empty(mountpoint)) { 303 zfs_error_aux(hdl, dgettext(TEXT_DOMAIN, 304 "directory is not empty")); 305 return (zfs_error_fmt(hdl, EZFS_MOUNTFAILED, 306 dgettext(TEXT_DOMAIN, "cannot mount '%s'"), mountpoint)); 307 }
Updated by Andrew Stormont almost 6 years ago
That seems perfectly reasonable to me. Here is the matching code from the zfs_mount function in zfs_vfsops.c, lines 1597-1604
mutex_enter(&mvp->v_lock); if ((uap->flags & MS_REMOUNT) == 0 && (uap->flags & MS_OVERLAY) == 0 && (mvp->v_count != 1 || (mvp->v_flag & VROOT))) { mutex_exit(&mvp->v_lock); return (SET_ERROR(EBUSY)); } mutex_exit(&mvp->v_lock);
Most of the other file systems have something similar in their mount functions too.
EDIT: I see that I have mentioned this before. The zfs command is working correctly.
Updated by Marcel Telka almost 6 years ago
Andrew Stormont wrote:
That seems perfectly reasonable to me. Here is the matching code from the zfs_mount function in zfs_vfsops.c, lines 1597-1604
[...]
Most of the other file systems have something similar in their mount functions too.
Note: The matching code in zfs_vfsops.c allows to mount in a case the mount point is not empty. There is no problem in zfs_vfsops.c.
Updated by Marcel Telka almost 6 years ago
The zfs(1m) man page does not suggest that the mount point directory must be empty. The man page says this about the mount subcommand:
zfs mount [-vO] [-o options] -a | filesystem Mounts ZFS file systems. Invoked automatically as part of the boot process. -o options An optional, comma-separated list of mount options to use temporarily for the duration of the mount. See the "Temporary Mount Point Properties" section for details. -O Perform an overlay mount. See mount(1M) for more information. -v Report mount progress. -a Mount all available ZFS file systems. Invoked automatically as part of the boot process. filesystem Mount the specified filesystem.
Updated by Marcel Telka almost 6 years ago
- Related to Bug #5841: NFSv3 writes underneath mounted filesystem to directory added
Updated by Marcel Telka over 5 years ago
- Related to Feature #997: RFE: ZFS filesystem dataset option to allow overlay mounts added