Project

General

Profile

Actions

Bug #16065

open

Panic in zfs_retzcbuf with smb2_read_zcopy enabled

Added by Gordon Ross 15 days ago. Updated 1 day ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
zfs - Zettabyte File System
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:
racktop:BSR-14478

Description

After #15615, got a panic that looks like this:


dmu_xuio_cnt+0xd(fffffe19bb3ddcc0)
zfs_retzcbuf+0x16(fffffdffccfba240, fffffe19bb3ddcc0, fffffdf3ec5323c8, ffffffffc01d8f90)
vhead_retzcbuf+0xdf(fffffdffccfba240, fffffe19bb3ddcc0, fffffdf3ec5323c8, ffffffffc01d8f90)
fop_retzcbuf+0x6b(fffffdffccfba240, fffffe19bb3ddcc0, fffffdf3ec5323c8, ffffffffc01d8f90)
smb_vop_retzcbuf+0x17(fffffdffccfba240, fffffe19bb3ddcc0, fffffdf3ec5323c8)
smb_fsop_retzcbuf+0x17(fffffe536b0ea998, fffffe19bb3ddcc0, fffffdf3ec5323c8)
smb_xuio_free+0x60(fffffe19bb3ddcc0)
smb2_read+0x504(ffffff544b887350)
smb2sr_work+0x58c(ffffff544b887350)
smb2_tq_work+0x73(ffffff544b887350)
taskq_d_svc_bucket+0x96(fffffdf486d6ab88, fffffdf3799d0578, fffffdf30bcba0e8)
taskq_d_thread+0x9a(fffffdf486d6ab88)
thread_start+0xb()


Files


Related issues

Related to illumos gate - Bug #15615: SMB2 read should use zero-copy VFS interfacesClosedToomas Soome

Actions
Actions #1

Updated by Gordon Ross 15 days ago

This panic happens because
XUIO_XUZC_PRIV(xuio) == NULL
when zfs_retzcbuf calls dmu_xuio_cnt.

How can that happen?

In zfs_reqzcbuf(), called with ioflg=UIO_READ,
if the function allows the buffer loan, it sets:
uio->uio_extflg = UIO_XUIO;
XUIO_XUZC_RW(xuio) = ioflag;
Setting that flag and returning success tells the caller they
should call VOP_RETZCBUF to "return" the loan later.
Note that zfs_reqzcbuf does NOT set:
XUIO_XUZC_PRIV(xuio) = (dmu_xuio_t *) priv;
That happens later, in dmu_xuio_init(), via ... zfs_read().

However, if (for example) zfs_read() returns an error
before it gets to where zfs_retzcbuf() is called, eg.
a mandatory lock prevents the read, or the read is for
data beyond the end of the file (it got truncated).

In those cases, zfs_read can return without calling
dmu_xuio_init(), so the private pointer is not set.
The ZFS layer has set: uio->uio_extflg = UIO_XUIO;
and therefore should be prepared for a call to
zfs_retzcbuf, but in the above cases, it does not
have the private data expected by zfs_retzcbuf.

One possible solution would be to have zfs_retzcbuf
"do nothing" when the private pointer is null.

As a short-term work-around, one can do:

mdb -kew 'smb2_read_zcopy/W0'

and/or add this line to /etc/system
set smb2_read_zcopy = 0

Actions #3

Updated by Gordon Ross 15 days ago

  • Related to Bug #15615: SMB2 read should use zero-copy VFS interfaces added
Actions #4

Updated by Gordon Ross 15 days ago

  • Description updated (diff)
Actions #5

Updated by Gordon Ross 3 days ago

  • Category set to zfs - Zettabyte File System
  • Status changed from New to In Progress
  • Assignee changed from Gordon Ross to Toomas Soome
Actions #6

Updated by Gordon Ross 3 days ago

  • External Bug set to racktop:BSR-14478
Actions #7

Updated by Electric Monk 2 days ago

  • Gerrit CR set to 3164
Actions #8

Updated by Gordon Ross 1 day ago

Reproduction method from Matt Barden:

I managed to reproduce this with a new smbtorture test [TBD] that issues a Create request with FILE_OVERWRITE while a Read beyond offset 0 is outstanding (on a 256k file that's reading > 128k of data - necessary for zfs_reqzcbuf to return 'success'). I used the following dtrace to help with the order the requests get processed (after using mdb to set dtrace_chill_max to 10 seconds):

dtrace -n 'fbt::smb_fsop_reqzcbuf:return { printf("%d\n", arg1)} fbt::smb_fsop_read:entry { chill(5000000000) ; printf("here")} fbt::zfs_reqzcbuf:entry { printf("here")} fbt::smb2_create:entry { chill(500000) ; printf("here")}' -w
Actions #9

Updated by Gordon Ross 1 day ago

  • Subject changed from Panic in smb2_read with zero copy buffer handling to Panic in zfs_retzcbuf with smb2_read_zcopy enabled
Actions

Also available in: Atom PDF