Project

General

Profile

Bug #4243

sdev_access() race leads to panic

Added by Robert Mustacchi about 7 years ago. Updated almost 7 years ago.

Status:
Resolved
Priority:
Normal
Category:
kernel
Start date:
2013-10-20
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

From the Joyent bug report:

We died in sdev_unlocked_access() as follows:

ffffff00c1a2b9e0 sdev_unlocked_access+0x2f(ffffff7b6f13b2b8, 80, ffffffdfc3ff6990)
ffffff00c1a2ba40 sdev_access+0x63(ffffff3262196d80, 80, 0, ffffffdfc3ff6990, 0)
ffffff00c1a2baa0 fop_access+0x8c(ffffff3262196d80, 80, 0, ffffffdfc3ff6990, 0)
ffffff00c1a2bae0 spec_access+0x44(ffffffff95bdf040, 80, 0, ffffffdfc3ff6990, 0)
ffffff00c1a2bb40 fop_access+0x8c(ffffffff95bdf040, 80, 0, ffffffdfc3ff6990, 0)
ffffff00c1a2bcf0 vn_openat+0x417(8047da5, 0, 2202, 0, ffffff00c1a2bd48, 0, 12, 0, 5)
ffffff00c1a2be60 copen+0x49e(ffd19553, 8047da5, 2202, 0)
ffffff00c1a2be90 openat64+0x2d(ffd19553, 8047da5, 201, 0)
ffffff00c1a2bec0 open64+0x2e(8047da5, 201, 0)
ffffff00c1a2bf10 _sys_sysenter_post_swapgs+0x149()

The problem here is that sdev_attrp is NULL. Yet if we look at sdev_access(), we clearly did not intend to get here:

        if (dv->sdev_attrvp) {
                ret = VOP_ACCESS(dv->sdev_attrvp, mode, flags, cr, ct);
        } else if (dv->sdev_attr) {
                rw_enter(&dv->sdev_contents, RW_READER);
                ret = sdev_unlocked_access(dv, mode, cr);

While the obvious explanation would be that we've failed to check for the zombie state after taking sdev_contents, this node isn't actually in the zombie state:

    sdev_state = 0x1
    sdev_flags = 0x1

It should also be noted that this node refers to /dev/null, and that 2 other threads hit this just slightly after we did. Finally, this is an old platform missing most of the sdev fixes; we're on joyent_20130111T180733Z. Nevertheless, this particular code has not changed so it seems quite possible that this race still exists.

--

What has most likely happened here is that we have recently created the attrvp. If we look here we'll see this:

> ffffff7b6f13b2b8::print sdev_node_t sdev_attrvp sdev_attr
sdev_attrvp = 0xffffff32616a7500
sdev_attr = 0

All the modifications of sdev_attr are done while holding the contents lock, but the same is not true for reading. This basically indicates missed synchronization. If we look at the a_time of the attrvp and compare it with the panic, they're about 7ms apart.

> ffffff7b6f13b2b8::print sdev_node_t sdev_attrvp->v_data | ::print znode_t z_atime
z_atime = [ 0x51c1fd96, 0x7c1766c ]
> 0x51c1fd96=Y
                2013 Jun 19 18:51:02    
> panic_hrestime::print timespec_t
{
    tv_sec = 2013 Jun 19 18:51:02
    tv_nsec = 0x75446ad
}
> 0x7c1766c-0x75446ad=K
                6d2fbf          
> 0x7c1766c-0x75446ad=D
                7155647         

So that's about a 7ms window. That's a lot of time, but we could have read some garbage given that we're outside the lock. The fix here is straightforward, we should be checking which of the attrs to use while holding the lock.

#1

Updated by Robert Mustacchi about 7 years ago

  • % Done changed from 90 to 100

Resolved in 368fc9412cc48af49761a1b8808d9c4079709391.

#2

Updated by Robert Mustacchi almost 7 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF