illumos gate

When smb encryption is enabled (encrypt=required), copying a large file from a share caused the transfer and machine to eventually hang. I broke into mdb and did a $<systemdump

> ::findleaks
findleaks: thread fffffe0bdd329be0's stack swapped out; false positives possible
findleaks: thread fffffe0bdcc8d800's stack swapped out; false positives possible
findleaks: thread fffffe0becf3a080's stack swapped out; false positives possible
findleaks: thread fffffe0bd5de1800's stack swapped out; false positives possible
findleaks: thread fffffe0bebd6b800's stack swapped out; false positives possible
findleaks: thread fffffe0bd525f800's stack swapped out; false positives possible
findleaks: thread fffffe0bed7a9400's stack swapped out; false positives possible
findleaks: thread fffffe0bdb7b6840's stack swapped out; false positives possible
findleaks: thread fffffe0bdcbe4460's stack swapped out; false positives possible
findleaks: thread fffffe0bdd179020's stack swapped out; false positives possible
findleaks: thread fffffe0bed37e3e0's stack swapped out; false positives possible
CACHE             LEAKED           BUFCTL CALLER
fffffe0bb842a008       1 fffffe0bcfcd82d8 AcpiOsAllocate+0x1c
fffffe0bb843b008       1 fffffe0beede51b0 allocb+0x50
fffffe0bb843a688       1 fffffe0becadf9b0 dblk_constructor+0x46
fffffe0bb842c348       5 fffffe0bf5568080 smb2_send_reply+0x7d
fffffe0bb842d348       3 fffffe0bee3216f0 smb2_send_reply+0x7d
fffffe0bb842c688       4 fffffe0c01a6d108 smb2_send_reply+0x7d
fffffe0bb842a9c8      11 fffffe0bf65ab670 smb2_send_reply+0x7d
fffffe0bb842d9c8       1 fffffe0bf5528c40 smb2_send_reply+0x7d
fffffe0bb842d008       9 fffffe0c02029dd8 smb2_send_reply+0x7d
fffffe0bb842d688       2 fffffe0bfcfd7b68 smb2_send_reply+0x7d
fffffe0bb842a9c8       1 fffffe0bf3934de0 smb2_send_reply+0x7d
fffffe0bb842d008       3 fffffe0c02029eb0 smb2_send_reply+0x7d
fffffe0bb842f348       3 fffffe0bf39530f0 smb2_send_reply+0x7d
fffffe0bb842c9c8       4 fffffe0bdc0f6970 smb2_send_reply+0x7d
fffffe0bb842c008       8 fffffe0bececd1f8 smb2_send_reply+0x7d
fffffe0bb842a9c8       4 fffffe0bfa018e00 smb2_send_reply+0x7d
fffffe0bb8434688       1 fffffe0c025e0618 smb2_send_reply+0x7d
fffffe0bb8437688   79821 fffffe0c05fffec8 smb2_send_reply+0x7d
fffffe0bb842c348       7 fffffe0c0201fd88 smb2_send_reply+0x7d
fffffe0bb8435008       1 fffffe0c0252e7c0 smb2_send_reply+0x7d
fffffe0bf78f3348       2 fffffe0c0c8a36c8 smb_mbuf_alloc+0x16
fffffe0bb842d348       1 fffffe0bdb110990 vn_make_ops+0x42
fffffe0bb842d348       1 fffffe0bdd67a978 vn_make_ops+0x42
------------------------------------------------------------------------
           Total   79895 buffers, 5885097440 bytes
> fffffe0c05fffec8$<bufctl_audit
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
fffffe0c05fffec8 fffffe0c06140000       2ffcd56c76 fffffe0010dccc20
                 fffffe0bb8437688 fffffe0bb9f05800                0
                 kmem_slab_alloc_impl+0x198
                 kmem_slab_alloc+0x62
                 kmem_cache_alloc+0x15b
                 kmem_alloc+0x4b
                 smb2_send_reply+0x7d
                 smb2sr_work+0x47f
                 smb2_tq_work+0x73
                 taskq_d_thread+0xbc
                 thread_start+0xb

The function (https://github.com/illumos/illumos-gate/blob/6937e379563aa5fe8a003acdcd316b89044dd8dd/usr/src/uts/common/fs/smbsrv/smb2_dispatch.c#L1470) initially didn't look like it leaked tmpbuf because smb_session_send(smb_session_t *session, uint8_t nbt_type, mbuf_chain_t *mbc) unconditionally frees mbc. But this entails calling mbc->chain->m_ext.ext_ref, and MBC_ATTACH_BUF(&enc_reply, tmpbuf, buflen) sets this to mclrefnoop, which does what its name says.

I tested a quick patch (attached) and copied the file successfully. Not sure if this is the nicest way to patch, you could use smb_mbuf_kmem_ref or something.