Project

General

Profile

Bug #5579

zfs mount is inconsistent with mount(1m)

Added by Marcel Telka over 5 years ago. Updated over 5 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
zfs - Zettabyte File System
Start date:
2015-01-31
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

The zfs mount command behaves differently than other mount operations. All filesystems, when mounted using the mount(1m) command, have no problem to mount even in a case the mountpoint directory is not empty. This is also documented in the mount(1m) man page:

       mount attaches a file system to the file system hierarchy at the
       mount_point, which is the pathname of a directory. If mount_point has
       any contents prior to the mount operation, these are hidden until the
       file system is unmounted.

Similar sentence is in mount man pages for particular filesystems, for example this is in mount_hsfs(1m):

       mount attaches an ISO 9660 filesystem (the High Sierra  file system,
       hsfs,  is  a  draft predecessor to ISO 9660, so the name reflects the
       filesystem's history) to  the  file  system hierarchy at the
       mount_point, which is the pathname of a directory. If mount_point has
       any contents prior to the mount operation, these are hidden until the
       file system is unmounted.

Actually, the zfs filesystem works similarly when mounted using the mount(1m) command, for example:

# ls -al 1
total 2
drwxr-xr-x 2 root root 3 jan 20 16:18 .
drwxr-xr-x 4 root root 4 jan 20 20:39 ..
-rw-r--r-- 1 root root 0 jan 20 16:18 a
# mount -F zfs rpool/export/test/1 1
# ls -al 1
total 1
drwxr-xr-x 2 root root 2 jan 20 16:02 .
drwxr-xr-x 4 root root 4 jan 20 20:39 ..
# umount 1
#

The only difference is when the zfs mount command is used:

# ls -al 1
total 2
drwxr-xr-x 2 root root 3 jan 20 16:18 .
drwxr-xr-x 4 root root 4 jan 20 20:39 ..
-rw-r--r-- 1 root root 0 jan 20 16:18 a
# zfs mount rpool/export/test/1
cannot mount '/export/test/1': directory is not empty
# zfs mount -O rpool/export/test/1
# ls -al 1
total 1
drwxr-xr-x 2 root root 2 jan 20 16:02 .
drwxr-xr-x 4 root root 4 jan 20 20:39 ..
#

As we see from the test, I had to use -O (overlay mount) to achieve the desired behavior.

I see few issues with the current zfs mount behavior:

  1. The -O flag is required in a case the mount directory is not empty. This is different behavior when compared to the mount(1m) command.
  2. When the -O flag is not used, the zfs(1m) produces error "directory is not empty", but according to the mount(1m) man page the error when the -O flag is not used (and is needed) should be "device busy".
  3. The zfs mount requires the -O flag for a case when we do not do the real overlay mount. This is confusing.

I'd expect that both zfs mount and mount(1m) commands behaves similarly, IOW, the zfs mount (without the -O flag) should not fail in our test case above.

The described difference between zfs mount and mount(1m) could cause some zfs filesystems are not mounted on boot.


Related issues

Related to illumos gate - Bug #5841: NFSv3 writes underneath mounted filesystem to directoryPending RTIMarcel Telka2015-04-14

Actions
Related to illumos gate - Feature #997: RFE: ZFS filesystem dataset option to allow overlay mountsNew2011-05-06

Actions
#1

Updated by Andrew Stormont over 5 years ago

Which filesystems did you try? In my experience both UFS and ZFS require the -O option to mount over the top of an another mount point (unless it's a remount) or a non empty directory.

I can point out where this is enforced in the code:

UFS: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/ufs/ufs_vfsops.c#L282
ZFS: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/zfs_vfsops.c#L1598

Both zfs(1m) and mount(1m) should be failing to mount over a non-empty directory. Maybe mount(1m) is doing something different like passing -o remount?

#2

Updated by Marcel Telka over 5 years ago

Andrew Stormont wrote:

Which filesystems did you try?

Several. The example with ZFS is in the Description. An example with UFS is here:

# mount | grep ^/mnt
# mkfile 100m /var/tmp/fs1
# lofiadm -a /var/tmp/fs1
/dev/lofi/1
# newfs /dev/lofi/1
newfs: construct a new file system /dev/rlofi/1: (y/n)? y
/dev/rlofi/1:   204600 sectors in 341 cylinders of 1 tracks, 600 sectors
        99,9MB in 22 cyl groups (16 c/g, 4,69MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,
 115232, 124832, 134432, 144032, 153632, 163232, 172832, 182432, 192032, 201632
# mount /dev/lofi/1 /mnt
# mkdir /mnt/a
# touch /mnt/a/b
# mkfile 100m /var/tmp/fs2
# lofiadm -a /var/tmp/fs2
/dev/lofi/2
# newfs /dev/lofi/2
newfs: construct a new file system /dev/rlofi/2: (y/n)? y
/dev/rlofi/2:   204600 sectors in 341 cylinders of 1 tracks, 600 sectors
        99,9MB in 22 cyl groups (16 c/g, 4,69MB/g, 2240 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432,
 115232, 124832, 134432, 144032, 153632, 163232, 172832, 182432, 192032, 201632
# mount /dev/lofi/2 /mnt/a
# ls -la /mnt/a
total 10
drwxr-xr-x 3 root root  512 mar  9 13:27 .
drwxr-xr-x 4 root root  512 mar  9 13:26 ..
drwx------ 2 root root 8192 mar  9 13:27 lost+found
# umount /mnt/a
# ls -la /mnt/a
total 2
drwxr-xr-x 2 root root 512 mar  9 13:26 .
drwxr-xr-x 4 root root 512 mar  9 13:26 ..
-rw-r--r-- 1 root root   0 mar  9 13:26 b
#
#3

Updated by Marcel Telka over 5 years ago

Because of this bug, here is a simple way how to make a system unbootable:

# zfs create -p rpool/test/a/b
# zfs umount rpool/test
# zfs mount rpool/test/a/b
# zfs mount -a
cannot mount '/rpool/test': directory is not empty
cannot mount '/rpool/test/a': directory is not empty
#

After the reboot the filesystem/local SMF service will go to maintenance and because of that the sshd won't start (and most of the services as well), so you just killed your remote server.

#4

Updated by Andrew Stormont over 5 years ago

I think the zfs command is doing the right thing here. The boot script should not be erroring out though so quickly. I have seen this myself when importing another rpool with -N option (which is not persistent - thought I'd filed a bug for it but can't find anything) and rebooting with it still imported.

I think these are all separate but related bugs.

#5

Updated by Marcel Telka over 5 years ago

The "zfs mount" is failing on non-empty mount points because of this piece of code in zfs_mount() (in libzfs):

294    /*
295     * Determine if the mountpoint is empty.  If so, refuse to perform the
296     * mount.  We don't perform this check if MS_OVERLAY is specified, which
297     * would defeat the point.  We also avoid this check if 'remount' is
298     * specified.
299     */
300    if ((flags & MS_OVERLAY) == 0 &&
301        strstr(mntopts, MNTOPT_REMOUNT) == NULL &&
302        !dir_is_empty(mountpoint)) {
303        zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
304            "directory is not empty"));
305        return (zfs_error_fmt(hdl, EZFS_MOUNTFAILED,
306            dgettext(TEXT_DOMAIN, "cannot mount '%s'"), mountpoint));
307    }
#6

Updated by Andrew Stormont over 5 years ago

That seems perfectly reasonable to me. Here is the matching code from the zfs_mount function in zfs_vfsops.c, lines 1597-1604

    mutex_enter(&mvp->v_lock);
    if ((uap->flags & MS_REMOUNT) == 0 &&
        (uap->flags & MS_OVERLAY) == 0 &&
        (mvp->v_count != 1 || (mvp->v_flag & VROOT))) {
        mutex_exit(&mvp->v_lock);
        return (SET_ERROR(EBUSY));
    }
    mutex_exit(&mvp->v_lock);

Most of the other file systems have something similar in their mount functions too.

EDIT: I see that I have mentioned this before. The zfs command is working correctly.

#7

Updated by Marcel Telka over 5 years ago

Andrew Stormont wrote:

That seems perfectly reasonable to me. Here is the matching code from the zfs_mount function in zfs_vfsops.c, lines 1597-1604

[...]

Most of the other file systems have something similar in their mount functions too.

Note: The matching code in zfs_vfsops.c allows to mount in a case the mount point is not empty. There is no problem in zfs_vfsops.c.

#8

Updated by Marcel Telka over 5 years ago

The zfs(1m) man page does not suggest that the mount point directory must be empty. The man page says this about the mount subcommand:

       zfs mount [-vO] [-o options] -a | filesystem
           Mounts ZFS file systems. Invoked automatically as part of the boot
           process.

           -o options
               An optional, comma-separated list of mount options to use
               temporarily for the duration of the mount. See the "Temporary
               Mount Point Properties" section for details.

           -O
               Perform an overlay mount. See mount(1M) for more information.

           -v
               Report mount progress.

           -a
               Mount all available ZFS file systems. Invoked automatically as
               part of the boot process.

           filesystem
               Mount the specified filesystem.
#9

Updated by Marcel Telka over 5 years ago

  • Related to Bug #5841: NFSv3 writes underneath mounted filesystem to directory added
#10

Updated by Marcel Telka over 5 years ago

  • Related to Feature #997: RFE: ZFS filesystem dataset option to allow overlay mounts added

Also available in: Atom PDF