Project

General

Profile

Bug #11928

rpcmod's clnt_cots can do zero-length kmem allocations

Added by Dan McDonald 16 days ago. Updated 7 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
nfs - NFS server and client
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

While testing NFS service in a zone using a DEBUG SmartOS, I discovered rpcmod was calling kmem_alloc with zero-length. While examining the clnt_cots.c code, I also found some unnecessary zalloc(), when a plain alloc() would do.

This bug is only scoped on $UTS/common/rpc/clnt_cots.c. A fuller audit of rpcmod is in order, but should be filed separately, after all distros have the functionality in #11927.


Related issues

Related to illumos gate - Bug #11927: Log, or optionally panic, on zero-length kmem allocationsNew

Actions

History

#1

Updated by Dan McDonald 16 days ago

  • Assignee set to Dan McDonald

NFSv4 hammering (callback or seems to trigger the zero-length allocations. Here's a list of stacks during one test:

T-0.000000000 addr=0 0
kmem_log_event+0x9b
kmem_alloc+0x17d
kmem_zalloc+0x10d
connmgr_get+0x49b
connmgr_wrapget+0x58
clnt_cots_kcallit+0x16b
rfs4_do_callback+0x97
rfs4_do_cb_recall+0xfc
do_recall+0x1af
thread_start+0xb

T-2967.420986776 addr=0 0
kmem_log_event+0x9b
kmem_alloc+0x17d
kmem_zalloc+0x10d
connmgr_get+0x49b
connmgr_wrapget+0x58
clnt_cots_kcallit+0x16b
rfs4_do_callback+0x97
rfs4_do_cb_recall+0xfc
do_recall+0x1af
thread_start+0xb

T-8424.755340132 addr=0 0
kmem_log_event+0x9b
kmem_alloc+0x17d
kmem_zalloc+0x10d
connmgr_get+0x49b
connmgr_wrapget+0x58
clnt_cots_kcallit+0x16b
rfs4_do_callback+0x97
rfs4_do_cb_recall+0xfc
do_recall+0x1af
thread_start+0xb

T-9263.085631357 addr=0 0
kmem_log_event+0x9b
kmem_alloc+0x17d
kmem_zalloc+0x10d
connmgr_get+0x49b
connmgr_wrapget+0x58
clnt_cots_kcallit+0x16b
rfs4_do_cb_null+0x11e
thread_start+0xb

T-16528.165042251 addr=0 0
kmem_log_event+0x9b
kmem_alloc+0x17d
kmem_zalloc+0x10d
connmgr_get+0x49b
connmgr_wrapget+0x58
clnt_cots_kcallit+0x16b
rfs4_do_cb_null+0x11e
thread_start+0xb

#2

Updated by Dan McDonald 15 days ago

Tested by re-running a nightly build using an nfs-zone-served zone on a DEBUG kernel. Before this fix, there were results from mdb's `::kmalog zerosized` on SmartOS, after this fix, none. See #11927 for the SmartOS functionality in question.

#3

Updated by Marcel Telka 15 days ago

  • Category set to nfs - NFS server and client
#4

Updated by Marcel Telka 15 days ago

From the kmem_alloc(9f) man page:

NOTES
       kmem_alloc(0, flag) always returns NULL.  kmem_free(NULL, 0) is legal.
#5

Updated by Marcel Telka 15 days ago

  • Related to Bug #11927: Log, or optionally panic, on zero-length kmem allocations added
#6

Updated by Dan McDonald 10 days ago

The output mentioned in comment 1 is from this mdb command from bug #11927:

::kmalog zerosized

Here are before (dump 3) and after (dump 4) tests:

root@larry (kebecloud) /var/crash/volatile]# echo '::kmalog zerosized' | mdb 3
mdb: warning: dump is from SunOS 5.11 joyent_20191022T221350Z; dcmds and macros may not match kernel implementation

T-0.000000000  addr=0  0
         kmem_log_event+0x9b
         kmem_alloc+0x17d
         kmem_zalloc+0x10d
         connmgr_get+0x49b
         connmgr_wrapget+0x58
         clnt_cots_kcallit+0x16b
         rfs4_do_cb_null+0x11e
         thread_start+0xb
[root@larry (kebecloud) /var/crash/volatile]# echo '::kmalog zerosized' | mdb 4
[root@larry (kebecloud) /var/crash/volatile]# 
#7

Updated by Electric Monk 7 days ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 4b6bffb4c4308c6219c095d4cf5bf96bb0970e04

commit  4b6bffb4c4308c6219c095d4cf5bf96bb0970e04
Author: Dan McDonald <danmcd@joyent.com>
Date:   2019-11-14T15:00:28.000Z

    11928 rpcmod's clnt_cots can do zero-length kmem allocations
    Reviewed by: Matthias Scheler <Matthias.Scheler@wdc.com>
    Reviewed by: Toomas Soome <tsoome@me.com>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Also available in: Atom PDF