Project

General

Profile

Actions

Bug #13739

closed

vnode reference leak in copen()

Added by Andy Fiddaman about 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
kernel
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

While testing #13380, I had a path in the global zone lofs mounted into a NGZ, and then passed to a bhyve guest using virtio-9p.
After running an rsync in the guest to write files to the filesystem, the zone would no longer shut down due to references on the lofs mount point vnode.

Having confirmed that nothing was accessing the directory by looking at zone processes and using pfiles, I changed to the directory and attempted to unmount it:

# dtrace -n 'fbt::lo_unmount:entry{li = (struct loinfo *)args[0]->vfs_data;printf("  li_refct = %d, rootvp = %d\n", li->li_refct, li->li_rootvp->v_count)}' -c "umount `pwd`" 
dtrace: description 'fbt::lo_unmount:entry' matched 1 probe
umount: /zones/deb/root/data/af/bhyve busy
dtrace: pid 28664 has exited
CPU     ID                    FUNCTION:NAME
  4  78768                 lo_unmount:entry   li_refct = 1, rootvp = 3611

The high reference count on the rootvp is a problem.

I used dtrace to watch holds and releases on this vnode while running rsync in the guest. The usual pattern was something like:

  3  11363                    copen:vn-hold bhyve vn-hold(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2545
  3  11371           lookuppnatcred:vn-hold bhyve vn-hold(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2546
  3  13564                lo_lookup:vn-hold bhyve vn-hold(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2547
  3  11306                  vn_rele:vn-rele bhyve vn-rele(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2546
  3  11306                  vn_rele:vn-rele bhyve vn-rele(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2545
  3  11306                  vn_rele:vn-rele bhyve vn-rele(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2544
  3  26906                     copen:return               @504

However, occasionally, when lookupnameat() fails, it is more like:

  3  11363                    copen:vn-hold bhyve vn-hold(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2545
  3  11371           lookuppnatcred:vn-hold bhyve vn-hold(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2546
  3  13564                lo_lookup:vn-hold bhyve vn-hold(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2547
  3  11306                  vn_rele:vn-rele bhyve vn-rele(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2546
  3  11306                  vn_rele:vn-rele bhyve vn-rele(fffffe2416de4080[/zones/deb/root/data/af/bhyve]) 2545
  3  31288               lookupnameat:entry ./bob/usr/lib/pymodules/python2.7/ndg_httpsclient-0.3.2.egg-info/namespace_packages.txt
  3  31289              lookupnameat:return                 2
  3  26906                     copen:return               @504

i.e. there is a reference leak in this case.

Actions #1

Updated by Electric Monk about 1 year ago

  • Gerrit CR set to 1429
Actions #2

Updated by Andy Fiddaman about 1 year ago

I tested this with a build and onu, and then running the workload that always previous reproduced the problem and confirmed that the vnode reference counts were the same before and after, and that the lofs filesystem could be unmounted after the test runs.

Actions #3

Updated by Dan McDonald about 1 year ago

Introduced in 2007 commit da6c28aaf62fa55f0fdb8004aa40f88f23bf53f0

Actions #4

Updated by Electric Monk about 1 year ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 99d240ec37397536063ecdea5644c8ef759c3e9a

commit  99d240ec37397536063ecdea5644c8ef759c3e9a
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2021-04-22T18:30:38.000Z

    13739 vnode reference leak in copen()
    Reviewed by: Jason King <jason.king@joyent.com>
    Reviewed by: Mike Zeller <mike.zeller@joyent.com>
    Reviewed by: Igor Kozhukhov <igor@dilos.org>
    Reviewed by: Andrew Stormont <andyjstormont@gmail.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF