illumos gate

Root cause:

We received very long write request:

> ::stack
vpanic()
segkp_fault+0x1fd(ffffff06f118be78, fffffffffbc31d80, ffffff00301aa000, 1000, 0, 1)
as_fault+0x31a(ffffff06f118be78, fffffffffbc30ac0, ffffff00301aa000, 1, 0, 1)
pagefault+0x96(ffffff00301aa000, 0, 1, 1)
trap+0x136f(ffffff003019ff20, ffffff00301aa000, 5)
0xfffffffffb8001d6()
0xfffffffffb853f8a()
uiocopy+0x16f(ffffff00e6e0c000, 20000, 1, ffffff00301a0580, ffffff00301a0268)
zfs_write+0x778(ffffff0b9c902d40, ffffff00301a0580, 0, ffffff0729797840, ffffff00301a05b0)
fop_write+0x5b(ffffff0b9c902d40, ffffff00301a0580, 0, ffffff0729797840, ffffff00301a05b0)
do_io+0x9e(2, ffffff0b9c902d40, ffffff00301a0580, 0, ffffff0729797840, ffffff00301a05b0)
rfs4_op_write+0x418(ffffff0726f0ccf0, ffffff0730f22e50, ffffff00301a0c00, ffffff00301a0640)
rfs4_compound+0x1e9(ffffff00301a0a00, ffffff00301a07d0, 0, ffffff00301a0c00, 0, ffffff00301a0808)
rfs4_dispatch+0x21d(fffffffffbd39670, ffffff00301a0c00, ffffff072674e200, ffffff00301a0a00)
common_dispatch+0x765(ffffff00301a0c00, ffffff072674e200, 2, 4, fffffffff8857376, fffffffffbd396c0)
rfs_dispatch+0x2d(ffffff00301a0c00, ffffff072674e200)
svc_getreq+0x1c1(ffffff072674e200, ffffff0700bc28c0)
svc_run+0xe0(ffffff0761362850)
svc_do_run+0x8e(1)
nfssys+0xf1(e, fed00fbc)
dtrace_systrace_syscall32+0xe4(e, fed00fbc, 9fb96b00, 26591760, 1, 0)
_sys_sysenter_post_swapgs+0x149()
> ffffff0726f0ccf0::print -t nfs_argop4 nfs_argop4_u.opwrite
WRITE4args nfs_argop4_u.opwrite = {
    stateid4 nfs_argop4_u.opwrite.stateid = {
        uint32_t seqid = 0
        char [12] other = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0' ]
    }
    offset4 nfs_argop4_u.opwrite.offset = 0xffffffff
    stable_how4 nfs_argop4_u.opwrite.stable = 0 (0)
    uint_t nfs_argop4_u.opwrite.data_len = 0xffffffff
    char *nfs_argop4_u.opwrite.data_val = 0
    mblk_t *nfs_argop4_u.opwrite.mblk = 0xffffff0700bc28c0
    struct clist *nfs_argop4_u.opwrite.rlist = 0
    CONN *nfs_argop4_u.opwrite.conn = 0
}
>

Please note data_len = 0xffffffff above. It means we received a write request for 2^32 - 1 bytes.

The problem is that various parts of the kernel treats so big numbers as signed, while they should be considered as unsigned (as specified in RFC 4506, Chapter 4.10. Variable-Length Opaque Data).

For example in rpcmod such big message should be discarded and never passed up to the nfs server, see mir_check_len(). But because 0xffffffff is evaluated as -1 the mir_check_len() is confused and accepted the message. Similar signed/unsigned issue is in xdrmblk_getmblk().

We need to go through all RPC and XDR related source code in kernel to make sure all lengths of the opaque data are treated as unsigned, not signed (this will be time consuming).

It looks like the problem is reproducible only with very long RPC messages (> 2GB) and this is not very common, because this issue seems to be in the kernel for very long time (very likely from day zero, so for about 20, maybe 30 years now) and nobody noticed it yet.

Code review: https://www.illumos.org/rb/r/53/