Project

General

Profile

Bug #7413

netstack leaks lead to blind netstack reuse

Added by Robert Mustacchi over 4 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Category:
kernel
Start date:
2016-09-27
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

I just took a quick look at thoth dump ef05877407cda414, which appears to be the dump from INC-780. We died here:

> *panic_thread::findstack -v
stack pointer for thread ffffff5bb34143e0: ffffff01fa0055b0
  ffffff01fa005900 0xffffff00505b7418()
  ffffff01fa005ad0 0x40()
  ffffff01fa005c20 udp_do_open+0x68(ffffff44e72bd0e0, 0, 0, ffffff01fa005ccc)
  ffffff01fa005c90 udp_create+0x69(2, 1, 0, ffffff436a07d420, ffffff436a07d2dc, ffffff01fa005ccc, ffffffff00000000, ffffff44e72bd0e0)
  ffffff01fa005d10 socket_init_common+0x1c0(ffffff436a07d2a8, 0, 0, ffffff44e72bd0e0)
  ffffff01fa005d40 so_init+0x25(ffffff436a07d2a8, 0, ffffff44e72bd0e0, 0)
  ffffff01fa005df0 socket_create+0xc2(2, 1, 0, 0, 0, 0, ffffff0100000001, ffffff44e72bd0e0, ffffff01fa005e48)
  ffffff01fa005eb0 so_socket+0x228(2, 1, 0, 0, 1)
  ffffff01fa005f10 _sys_sysenter_post_swapgs+0x153()

which is in udp_do_open(), immediately after the call to netstack_find_by_cred() near the top of the function. Sure enough, %rax is 0, so netstack_find_by_cred() returned NULL:

> ::panicinfo ! grep rax
             rax                0

Note that the zone_t in the passed cred_t is valid and "running":

> ffffff44e72bd0e0::print cred_t cr_zone | ::zone
            ADDR   ID STATUS        NAME                PATH
ffffff448c8ecd40 5151 running       666a52ea-0662-4a... /zones/666a52ea-0662-...

> ffffff44e72bd0e0::print cred_t cr_zone->zone_status
cr_zone->zone_status = 4 (ZONE_IS_RUNNING)

and it's definitely curthread's zone, so we should enter netstack_get_current(). But that function returns NULL if netstack_flags has NSF_CLOSING (0x2) set, and sure enough:

> ffffff44e72bd0e0::print cred_t cr_zone->zone_netstack->netstack_flags
cr_zone->zone_netstack->netstack_flags = 0x12

This appears to be a race between zone shutdown and UDP socket open. I'm not sure what's supposed to prevent this (i.e., whether we should be handling NULL here, or whether the caller has violated an invariant by invoking udp_do_open() with a netstack that's shutting down).


The above analysis all seems correct, but I am not sure we can conclude this is a race between bootup and shutdown. Here is what is running in the zone:

> ffffff448c8ecd40::print zone_t zone_zsched | ::ptree
fffffffffbc304c0  sched
     ffffff431d99a010  init
          ffffffa563bc5010  zsched
               ffffff5c05f04048  init
                    fffffff8d3db8010  svc.configd
                    ffffff5f7acc2020  svc.startd
                         ffffff46b7c85008  sulogin
                              ffffffb90d21d010  sulogin
                         fffffff7a703d050  net-ipmgmt
                              ffffff46d657c008  ipmgmtd
                                   fffffffd28b33048  ipmgmtd

So, it does look like the zone is just boot. However, here is the zone status and all of the zsd flags:

> ffffff448c8ecd40::print zone_t zone_status          
zone_status = 4 (ZONE_IS_RUNNING)
> ffffff448c8ecd40::walk zsd | ::print struct zsd_entry zsd_flags
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0
zsd_flags = 0
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0
zsd_flags = 0
zsd_flags = 0x4
zsd_flags = 0
zsd_flags = 0
zsd_flags = 0
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0x4
zsd_flags = 0
zsd_flags = 0
zsd_flags = 0

It doesn't look like we're actually trying to shutdown this zone, although it is correct that netstack_zone_shutdown looks like the only place where the netstack's NSF_ZONE_SHUTDOWN flag is set.


In taking a look at the dumps and the code, it seemed that the only way that this state was possible was if a netstack is destroyed but never inactivated due to a held reference count – and then the zone ID wraps. Indeed, in this case (a netstack that wasn't freed long enough for the zone ID to wrap), there seems to be nothing to protect us from this failure: the ASSERT that should fire in `netstack_zone_create` that we should "never find a pre-existing exclusive stack" won't fire (as an aside, this should clearly be a VERIFY), and we will end up dying shortly later having a netstack that we can't actually find.

I explored this hypothesis by deliberately inducing a leak (by manually increasing the reference count on a netstack), and then inducing zone ID wraparound. (With the default MAX_ZONEID of 9999, this can take a while – I manually lowered this in zone_init() to much more readily attain wraparound.)

The resulting panic upon (re)allocating the leaked netstack matches the observed panic:

> ::status
debugging crash dump vmcore.23 (64-bit) from headnode
operating system: 5.11 joyent_20160820T225931Z (i86pc)
image uuid: (not set)
panic message: 
BAD TRAP: type=e (#pf Page fault) rp=ffffff000c1b2ad0 addr=40 occurred in module
 "ip" due to a NULL pointer dereference
dump content: kernel pages only
> $c
udp_do_open+0x68(ffffff0318308010, 0, 0, ffffff000c1b2ccc)
udp_create+0x69(2, 1, 0, ffffff034ad4e6f8, ffffff034ad4e5b4, ffffff000c1b2ccc)
socket_init_common+0x1c0(ffffff034ad4e580, 0, 0, ffffff0318308010)
so_init+0x25(ffffff034ad4e580, 0, ffffff0318308010, 0)
socket_create+0xc2(2, 1, 0, 0, 0, 0)
so_socket+0x228(2, 1, 0, 0, 1)
_sys_sysenter_post_swapgs+0x153()

Of course, this is only possible if zone IDs have wrapped – which is only itself possible if we have created over 9999 zones over the uptime of the system. In all the dumps I looked at, this was the case; for example, in thoth dump ef05877407cda414:

> *zoneid_space::print vmem_t vm_kstat.vk_alloc.value.i32 vm_kstat.vk_free.value.i32
vm_kstat.vk_alloc.value.i32 = 0x1728a
vm_kstat.vk_free.value.i32 = 0x1720b

This, then, creates two possibilities: a reference was either held a long, long time after zone destruction (seems unlikely) or a reference was leaked (seems much more likely).

Assuming a leak, it would seem that in our reproducible case, more than one netstack would have been leaked. Looking at thoth dump 4ec0d1821a9dd25b:

> *netstack_head::list netstack_t netstack_next  | ::printf "%p 0x%02x %d %d\n" netstack_t . netstack_flags netstack_stackid netstack_refcnt
ffffff257a55ce00 0x00 0 76
ffffff26675cf140 0x00 1 7
ffffff26675cf3c0 0x00 2 41
ffffff26664b6300 0x00 3 45
ffffff266570fcc0 0x00 4 53
ffffff266657e640 0x00 5 1050
ffffff2662ff56c0 0x00 6 10
ffffff266ae81cc0 0x00 7 8
ffffff266216edc0 0x00 8 75
ffffff2661f81040 0x00 9 207
ffffff2663b37c80 0x00 10 15
ffffff2661f81900 0x00 11 6
ffffff26664b6d00 0x00 12 18
ffffff266373ae40 0x00 13 19
ffffff266906ca00 0x00 14 34
ffffff2667b573c0 0x00 15 25
ffffff266b996cc0 0x00 16 29
ffffff2667c5ca40 0x00 17 21
ffffff266974d7c0 0x00 18 11
ffffff266927be40 0x00 19 35
ffffff26674a6bc0 0x00 20 13
ffffff26620a8d00 0x00 21 43
ffffff2669be9580 0x00 22 19
ffffff266779a080 0x00 23 23
ffffff266a61ed00 0x00 24 10
ffffff2661f81680 0x00 25 12
ffffff2664f543c0 0x00 26 15
ffffff266974dcc0 0x00 27 13
ffffff266d673bc0 0x00 28 28
ffffff266a71fb80 0x00 29 9
ffffff26672e8180 0x00 30 20
ffffff26c55db540 0x00 31 1
ffffff36aaa2fb40 0x02 4536 1
ffffff5f15eaa040 0x02 4537 1
ffffff32fc9abdc0 0x02 4538 1
ffffff3304fc3080 0x02 4539 1
ffffff26a7466a00 0x02 4540 1
ffffff2d6da3f400 0x02 4541 1
ffffff3887f15d00 0x02 4542 1
ffffff33502e1540 0x02 4543 1
ffffff36a37fb900 0x02 4544 1
ffffff334aa8a440 0x02 4545 1
ffffff2ef6ee82c0 0x02 4546 1
ffffff33577f0a80 0x02 4547 1
ffffff5f717022c0 0x02 4548 1
ffffff36a79a7040 0x02 4549 1
ffffff3306fb37c0 0x02 4550 1          
ffffff8aff88a000 0x02 4551 1
ffffff2ef806f8c0 0x02 4553 1
ffffff2b50779a40 0x02 4578 1
ffffff2ed236b640 0x02 4579 1
ffffff3469ab3500 0x02 8255 1
ffffff36a3b4c900 0x02 8256 1
ffffff3307350540 0x02 8257 1
ffffff8b0b7eca00 0x02 8258 1
ffffff26628e2180 0x02 8259 1
ffffff3459e6bbc0 0x02 8260 1
ffffff2de8986b40 0x02 8261 1
ffffff36a0fd6500 0x02 8262 1
ffffff2f9cc426c0 0x02 8263 1
ffffff3314ba7d00 0x02 8264 1
ffffff7ac1e68040 0x02 8265 1
ffffff7bba1f3e00 0x02 8266 1
ffffff36a5101e40 0x02 8267 1
ffffff5f3aaa81c0 0x02 8268 1
ffffff8aff88aa00 0x02 8269 1
ffffff297af5a680 0x02 8270 1
ffffff3352f97800 0x02 8272 1
ffffff2c14c72280 0x02 8297 1
ffffff334e9d7e00 0x02 8298 1
ffffff26f4dac1c0 0x00 4328 1
ffffff337f235cc0 0x00 4378 4
ffffff3469851400 0x00 4380 4
ffffff2fc5246a00 0x00 4381 4
ffffff26bbeb2040 0x00 4382 4
ffffff33508bcd00 0x00 4383 4
ffffff369036fa40 0x00 4384 4
ffffff8ad99f7c80 0x00 4385 4
ffffff334fb55cc0 0x00 4386 4
ffffff5f8bac9040 0x00 4387 4
ffffff5f7157f280 0x00 4388 4
ffffff5f893dd140 0x00 4389 4
ffffff36a5ebdcc0 0x00 4390 4
ffffff3356168bc0 0x00 4391 4
ffffff334e5be1c0 0x00 4392 4
ffffff5ec4dfd640 0x00 4393 4
ffffff32fc70cb40 0x00 4394 4
ffffff39363c7d00 0x00 4395 4
ffffff36aa419300 0x00 4396 4
ffffff7ac98bd6c0 0x00 4397 4
ffffff5f6a5addc0 0x00 4398 4
ffffff330a04ce00 0x00 4400 4
ffffff2f793692c0 0x00 4440 1
ffffff8b45f47d00 0x00 4441 1
ffffff5f3c0ba780 0x00 4443 1
ffffff330d46f300 0x00 4445 1
ffffff280e954180 0x00 4446 4
ffffff36a44ecb40 0x00 4447 4
ffffff2b506b58c0 0x00 4448 4
ffffff2f9cc42e40 0x00 4449 4
ffffff2cade45e00 0x00 4450 1
ffffff330a2b5900 0x00 4451 4
ffffff36a80e7cc0 0x00 4453 4
ffffff334ca5a940 0x00 4454 4
ffffff5f3bf69bc0 0x00 4452 1
ffffff36a1989780 0x00 4455 4
ffffff2fc524c7c0 0x00 4456 4
ffffff26ba200a40 0x00 4457 4
ffffff330a200b40 0x00 4458 4
ffffff2f67f9e940 0x00 4460 4
ffffff280ea25580 0x00 4461 4
ffffff3935b8ba00 0x00 4462 4
ffffff8b4d160280 0x00 4463 4
ffffff36a42ce580 0x00 4464 4
ffffff3306d08a00 0x00 4465 4
ffffff5f8bdf4000 0x00 4466 4
ffffff280e954400 0x00 4467 4
ffffff7bba1f37c0 0x00 4468 4
ffffff3301d7e8c0 0x00 4469 4
ffffff2df926f8c0 0x00 4470 4
ffffff2cadd76d00 0x00 4471 4
ffffff5f8bdf9d00 0x00 4472 4
ffffff79675e6d00 0x00 4473 4
ffffff36a77ec140 0x00 4474 4
ffffff5f88ca9e40 0x00 4475 4
ffffff36a7ef8bc0 0x00 4476 4
ffffff5f8b9687c0 0x00 4477 4
ffffff3305c1e540 0x00 4478 4
ffffff36a98b8b40 0x00 4479 4
ffffff36a1e4eb80 0x00 4480 4
ffffff26849876c0 0x00 4481 4
ffffff33573a0e00 0x00 4482 4
ffffff28f56acbc0 0x00 4483 4
ffffff33f48d96c0 0x00 4484 4
ffffff8b4d22d2c0 0x00 4485 4
ffffff36729e9900 0x00 4486 4
ffffff5f8920a180 0x00 4487 4
ffffff28d0f39b80 0x00 4488 4
ffffff2c1482d000 0x00 4489 4
ffffff367f9c1800 0x00 4491 4
ffffff33949136c0 0x00 4490 4
ffffff7ac9573080 0x00 4492 4
ffffff5888730300 0x00 4493 4
ffffff330e794580 0x00 4494 4
ffffff8b4d980900 0x00 4495 4
ffffff33577ba080 0x00 4496 4
ffffff7ac976b400 0x00 4497 4
ffffff589d335c80 0x00 4498 4
ffffff26bbb297c0 0x00 4500 4
ffffff36a5dd4280 0x00 4499 4
ffffff26ba2007c0 0x00 4502 4
ffffff3cd8877a00 0x00 4501 4
ffffff3cd8b1b140 0x00 4503 4
ffffff33147a33c0 0x00 4504 4
ffffff3888024280 0x00 4505 4
ffffff335566d580 0x00 4506 4
ffffff5e9c212040 0x00 4507 4
ffffff345a1308c0 0x00 4508 4
ffffff36a9224300 0x00 4509 4
ffffff435363e8c0 0x00 4510 4
ffffff8ad9bcde40 0x00 4511 4
ffffff33557703c0 0x00 4512 4
ffffff8b4d0576c0 0x00 4513 4
ffffff3cd7bc4b80 0x00 4514 4
ffffff33f4f771c0 0x00 4515 4
ffffff280e762d00 0x00 4516 4
ffffff7ac2162540 0x00 4517 4
ffffff3935e86140 0x00 4519 4
ffffff3352f976c0 0x00 4520 4
ffffff5e9c212680 0x00 4521 4
ffffff8b56568c80 0x00 4522 4
ffffff4ace613e00 0x00 4523 4
ffffff33f513b900 0x00 4525 4
ffffff2ef806c300 0x00 4526 4
ffffff36a9abc000 0x00 4528 4
ffffff36a2c31540 0x00 4529 4
ffffff392a54bd00 0x00 4530 4
ffffff7c475848c0 0x00 4531 4
ffffff2f3ff293c0 0x00 4532 3
ffffff2f3ff29b40 0x00 4533 3
ffffff2d25f2eb80 0x00 4534 1
ffffff899bbdac80 0x00 4535 4

The netstacks with flags of 0 indicate allocated netstacks – but note how many netstacks have NSF_CLOSING set. There are two ranges of netstacks IDs that represent leaked netstacks: from 4536 to 4551, then 4553, then 4578 to 4579, from 8255 to 8270, 8272, and finally 8297 to 8298. (The symmetry here is odd: once netstacks start to be leaked, fifteen are leaked in a row, one is not leaked, then next is leaked, 25 aren't leaked and then a final two are leaked.)

This data indicates that leaks are happening long, long before panics. To explore the leak, I wrote a monster D script, which is also attached as OS-4810-leak.d:

#pragma D option quiet
#pragma D option nspec=256
#pragma D option specsize=256k

BEGIN
{
    stackid = 0;
}

netstack_zone_create:return
/(this->ns = (netstack_t *)arg1)->netstack_stackid != 0/
{
    specs[this->ns->netstack_stackid] = speculation();
    destroyed[this->ns->netstack_stackid] = 0;
}

netstack_hold:entry
/specs[args[0]->netstack_stackid]/
{
    speculate(specs[args[0]->netstack_stackid]);
    printf("\n%d HOLD %p %d", timestamp, args[0],
        args[0]->netstack_refcnt);
    stack();
}

netstack_rele:entry
/specs[args[0]->netstack_stackid]/
{
    speculate(specs[args[0]->netstack_stackid]);
    printf("\n%d RELE %p %d", timestamp, args[0],
        args[0]->netstack_refcnt);
    stack();
}

netstack_stack_inactive:entry
/specs[args[0]->netstack_stackid]/
{
    discard(specs[args[0]->netstack_stackid]);
    specs[args[0]->netstack_stackid] = 0;
    destroyed[args[0]->netstack_stackid] = 0;
}

netstack_zone_destroy:entry
/(this->ns = (netstack_t *)arg1) != NULL &&
    specs[this->ns->netstack_stackid]/
{
    destroyed[this->ns->netstack_stackid] = timestamp;
    speculate(specs[this->ns->netstack_stackid]);
    printf("\n%d DESTROY %p", timestamp, this->ns);
}

tick-1ms
{
    stackid = (stackid + 1) % 10000;
}

tick-1ms
/specs[stackid] && destroyed[stackid] &&
    (timestamp - destroyed[stackid]) / 1000000000 > 10/
{
    /*
     * We have had a reference hold that lived at least 10 seconds post
     * zone destruction -- dump it.
     */
    commit(specs[stackid]);
    specs[stackid] = 0;
    destroyed[stackid] = 0;
}

The output of this should be redirected to a file:

# dtrace -s ./OS-4810-leak.d > OS-4810-leak.out 2> OS-4810-err.out &

If that file ever has non-zero output (which it may/should long, long before a panic) it should point to a leak. (We should also watch for speculative drops; the buffer sizes and/or the number of speculative buffers may need to be increased.)

Looking at the .2 dump, the failed netstack is 0xffffff290854e7c0; I flattened the stacks

cat OS-4810-leak.2.out | awk '{ if ($1 ~ /^[0-9]/) { printf("\n"); printf("%s ", $0) } else { printf("%s:", $1); } }' | grep ffffff290854e7c0 | sort -n 
44510509702718 HOLD ffffff290854e7c0 1 genunix`netstack_next+0xb2:ip`ip_dce_reclaim+0x38:ip`dce_reclaim_worker+0xf0:unix`thread_start+0x8::
44510509705858 RELE ffffff290854e7c0 2 ip`ip_dce_reclaim+0x30:ip`dce_reclaim_worker+0xf0:unix`thread_start+0x8::
44534515273117 HOLD ffffff290854e7c0 1 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44534515279321 RELE ffffff290854e7c0 2 genunix`stropen+0x407:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44534817293586 HOLD ffffff290854e7c0 1 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44534817301183 RELE ffffff290854e7c0 2 genunix`stropen+0x407:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44535820609400 HOLD ffffff290854e7c0 1 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44535820615559 RELE ffffff290854e7c0 2 genunix`stropen+0x407:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44536820795832 HOLD ffffff290854e7c0 1 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44536820802615 RELE ffffff290854e7c0 2 genunix`stropen+0x407:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537833137154 HOLD ffffff290854e7c0 1 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537833143431 RELE ffffff290854e7c0 2 genunix`stropen+0x407:sockfs`sotpi_init+0xa1:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537870678601 HOLD ffffff290854e7c0 1 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537870688645 HOLD ffffff290854e7c0 2 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537870692382 RELE ffffff290854e7c0 3 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537870703239 HOLD ffffff290854e7c0 2 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537870705668 HOLD ffffff290854e7c0 3 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537870707772 RELE ffffff290854e7c0 4 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871186985 HOLD ffffff290854e7c0 3 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`rts_open+0x32:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871192097 HOLD ffffff290854e7c0 4 ip`ipcl_conn_create+0x57:ip`rts_open+0x56:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871194892 RELE ffffff290854e7c0 5 ip`rts_open+0x69:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871252611 RELE ffffff290854e7c0 4 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537871260192 RELE ffffff290854e7c0 3 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537871264822 RELE ffffff290854e7c0 2 ip`ipcl_conn_destroy+0x1bb:ip`rts_common_close+0x47:ip`rts_close+0x1e:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537871283097 HOLD ffffff290854e7c0 1 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871285728 HOLD ffffff290854e7c0 2 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871287731 RELE ffffff290854e7c0 3 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871292225 HOLD ffffff290854e7c0 2 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871294388 HOLD ffffff290854e7c0 3 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871296254 RELE ffffff290854e7c0 4 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871320643 HOLD ffffff290854e7c0 3 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`rts_open+0x32:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871323573 HOLD ffffff290854e7c0 4 ip`ipcl_conn_create+0x57:ip`rts_open+0x56:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871325426 RELE ffffff290854e7c0 5 ip`rts_open+0x69:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537871472600 RELE ffffff290854e7c0 4 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537871479777 RELE ffffff290854e7c0 3 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537871483917 RELE ffffff290854e7c0 2 ip`ipcl_conn_destroy+0x1bb:ip`rts_common_close+0x47:ip`rts_close+0x1e:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537874702871 HOLD ffffff290854e7c0 1 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537874713606 HOLD ffffff290854e7c0 2 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537874716801 RELE ffffff290854e7c0 3 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537874724628 HOLD ffffff290854e7c0 2 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537874726329 HOLD ffffff290854e7c0 3 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537874728163 RELE ffffff290854e7c0 4 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537875134834 HOLD ffffff290854e7c0 3 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`rts_open+0x32:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537875139153 HOLD ffffff290854e7c0 4 ip`ipcl_conn_create+0x57:ip`rts_open+0x56:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537875141378 RELE ffffff290854e7c0 5 ip`rts_open+0x69:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537875326331 HOLD ffffff290854e7c0 4 ip`ill_lookup_on_name+0x13d:ip`ipif_lookup_on_name+0xc7:ip`ip_sioctl_addif+0x31a:ip`ip_process_ioctl+0x28d:ip`ip_wput_nondata+0x1ba:ip`ip_helper_wput+0x23:unix`putnext+0x225:genunix`strdoioctl+0x2fd:genunix`strioctl+0x348:genunix`ldi_ioctl+0x6c:ip`udp_ioctl+0x64:sockfs`so_ioctl+0x1b9:sockfs`socket_ioctl+0x27:sockfs`socket_vop_ioctl+0x29:genunix`fop_ioctl+0x55:genunix`ioctl+0x9b:unix`_sys_sysenter_post_swapgs+0x153::
44537875405943 HOLD ffffff290854e7c0 5 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875420873 RELE ffffff290854e7c0 6 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875435354 HOLD ffffff290854e7c0 5 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875437634 HOLD ffffff290854e7c0 5 ip`ill_lookup_on_name+0x13d:ip`ipif_lookup_on_name+0xc7:ip`ip_sioctl_addif+0x31a:ip`ip_process_ioctl+0x28d:ip`ip_wput_nondata+0x1ba:ip`ip_helper_wput+0x23:unix`putnext+0x225:genunix`strdoioctl+0x2fd:genunix`strioctl+0x348:genunix`ldi_ioctl+0x6c:ip`udp_ioctl+0x64:sockfs`so_ioctl+0x1b9:sockfs`socket_ioctl+0x27:sockfs`socket_vop_ioctl+0x29:genunix`fop_ioctl+0x55:genunix`ioctl+0x9b:unix`_sys_sysenter_post_swapgs+0x153::
44537875465903 RELE ffffff290854e7c0 7 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875488595 HOLD ffffff290854e7c0 6 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875501683 RELE ffffff290854e7c0 7 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875521465 HOLD ffffff290854e7c0 6 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875527204 RELE ffffff290854e7c0 7 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537875817496 HOLD ffffff290854e7c0 6 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875821686 HOLD ffffff290854e7c0 7 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875823756 RELE ffffff290854e7c0 8 ip`udp_do_open+0xad:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875846485 HOLD ffffff290854e7c0 7 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875848661 RELE ffffff290854e7c0 8 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875876199 RELE ffffff290854e7c0 7 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_tpi_close+0x2c:genunix`qdetach+0x9b:genunix`strclose+0x2c1:specfs`device_close+0xb9:specfs`spec_close+0x17b:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537875894571 HOLD ffffff290854e7c0 6 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875896595 HOLD ffffff290854e7c0 7 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875898222 RELE ffffff290854e7c0 8 ip`udp_do_open+0xad:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875907023 HOLD ffffff290854e7c0 7 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875908799 RELE ffffff290854e7c0 8 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537875921413 RELE ffffff290854e7c0 7 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_tpi_close+0x2c:genunix`qdetach+0x9b:genunix`strclose+0x2c1:specfs`device_close+0xb9:specfs`spec_close+0x17b:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537876561397 HOLD ffffff290854e7c0 6 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537876570762 RELE ffffff290854e7c0 7 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537876591470 HOLD ffffff290854e7c0 6 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`rts_open+0x32:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537876593088 HOLD ffffff290854e7c0 6 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537876596587 HOLD ffffff290854e7c0 8 ip`ipcl_conn_create+0x57:ip`rts_open+0x56:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537876597394 RELE ffffff290854e7c0 9 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537876599202 RELE ffffff290854e7c0 8 ip`rts_open+0x69:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537876727484 HOLD ffffff290854e7c0 7 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537876732844 RELE ffffff290854e7c0 8 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537876753367 HOLD ffffff290854e7c0 7 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537876759688 RELE ffffff290854e7c0 8 ip`ipcl_conn_destroy+0x1bb:ip`rts_common_close+0x47:ip`rts_close+0x1e:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537876761204 RELE ffffff290854e7c0 8 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877454909 HOLD ffffff290854e7c0 6 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537877459810 HOLD ffffff290854e7c0 7 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537877462470 RELE ffffff290854e7c0 8 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
44537877592252 HOLD ffffff290854e7c0 7 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877599081 RELE ffffff290854e7c0 8 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877621212 HOLD ffffff290854e7c0 7 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877622478 RELE ffffff290854e7c0 8 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537877628569 RELE ffffff290854e7c0 7 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877641661 HOLD ffffff290854e7c0 6 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877647054 RELE ffffff290854e7c0 7 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877666722 HOLD ffffff290854e7c0 6 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537877670765 RELE ffffff290854e7c0 7 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44537879418262 RELE ffffff290854e7c0 6 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537879435075 RELE ffffff290854e7c0 5 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537879457167 RELE ffffff290854e7c0 4 ip`ipcl_conn_destroy+0x1bb:ip`rts_common_close+0x47:ip`rts_close+0x1e:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
44537918099496 HOLD ffffff290854e7c0 3 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537918105622 RELE ffffff290854e7c0 4 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537918187245 HOLD ffffff290854e7c0 3 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537918189467 RELE ffffff290854e7c0 4 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
44537975533511 HOLD ffffff290854e7c0 3 genunix`netstack_get_current+0x44:lx_brand`lx_fix_netstack+0x1d:lx_brand`lx_brandsys+0xc5f:genunix`brandsys+0x6d:unix`sys_syscall+0x196::
44540769137831 HOLD ffffff290854e7c0 4 genunix`netstack_find_by_zoneid+0x4c:genunix`netstack_find_by_cred+0x3f:ip`udp_do_open+0x68:ip`udp_open+0xd8:ip`udp_openv6+0x27:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a::
44540769148609 HOLD ffffff290854e7c0 5 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_open+0xd8:ip`udp_openv6+0x27:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237::
44540769151158 RELE ffffff290854e7c0 6 ip`udp_do_open+0xad:ip`udp_open+0xd8:ip`udp_openv6+0x27:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769194650 HOLD ffffff290854e7c0 5 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769197508 RELE ffffff290854e7c0 6 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769259713 HOLD ffffff290854e7c0 5 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769272299 RELE ffffff290854e7c0 6 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769311205 HOLD ffffff290854e7c0 5 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769324537 RELE ffffff290854e7c0 6 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769392127 RELE ffffff290854e7c0 5 ip`ill_delete_tail+0x292:ip`ip_sioctl_removeif+0x31f:ip`ip_process_ioctl+0x102:ip`ip_wput_nondata+0x1ba:ip`udp_wput_iocdata+0x4c:ip`udp_wput_other+0x100:ip`udp_wput+0xd7:unix`put+0x1b5:ip`mi_copyin+0xed:ip`ip_sioctl_copyin_setup+0x305:ip`ip_wput_nondata+0x25b:ip`udp_wput_other+0x73:ip`udp_wput+0xd7:unix`putnext+0x225:genunix`strdoioctl+0x3d2:genunix`strioctl+0x1e5a:genunix`ldi_ioctl+0x6c:ip`ip_ioctl+0x44:ip`ip_loopback_removeif+0x78:ip`ip_interface_cleanup+0xda::
44540769408991 HOLD ffffff290854e7c0 4 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769417793 RELE ffffff290854e7c0 5 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769419437 RELE ffffff290854e7c0 4 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_tpi_close+0x2c:genunix`qdetach+0x9b:genunix`strclose+0x2c1:specfs`device_close+0xb9:specfs`spec_close+0x17b:genunix`fop_close+0x61:genunix`ldi_close+0x93:ip`ip_interface_cleanup+0xfa:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769441179 HOLD ffffff290854e7c0 3 genunix`netstack_find_by_zoneid+0x4c:genunix`netstack_find_by_cred+0x3f:ip`udp_do_open+0x68:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a::
44540769443852 HOLD ffffff290854e7c0 4 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237::
44540769445843 RELE ffffff290854e7c0 5 ip`udp_do_open+0xad:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769461907 HOLD ffffff290854e7c0 4 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769464259 RELE ffffff290854e7c0 5 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769497240 HOLD ffffff290854e7c0 4 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769503118 RELE ffffff290854e7c0 5 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769574230 HOLD ffffff290854e7c0 4 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769579357 RELE ffffff290854e7c0 5 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769584848 HOLD ffffff290854e7c0 4 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769587311 RELE ffffff290854e7c0 5 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769599178 RELE ffffff290854e7c0 4 ip`ill_delete_tail+0x292:ip`ip_sioctl_removeif+0x31f:ip`ip_process_ioctl+0x102:ip`ip_wput_nondata+0x1ba:ip`udp_wput_iocdata+0x4c:ip`udp_wput_other+0x100:ip`udp_wput+0xd7:unix`put+0x1b5:ip`mi_copyin+0xed:ip`ip_sioctl_copyin_setup+0x305:ip`ip_wput_nondata+0x25b:ip`udp_wput_other+0x73:ip`udp_wput+0xd7:unix`putnext+0x225:genunix`strdoioctl+0x3d2:genunix`strioctl+0x1e5a:genunix`ldi_ioctl+0x6c:ip`ip_ioctl+0x44:ip`ip_loopback_removeif+0x78:ip`ip_interface_cleanup+0xda::
44540769618034 RELE ffffff290854e7c0 3 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_tpi_close+0x2c:genunix`qdetach+0x9b:genunix`strclose+0x2c1:specfs`device_close+0xb9:specfs`spec_close+0x17b:genunix`fop_close+0x61:genunix`ldi_close+0x93:ip`ip_interface_cleanup+0xfa:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
44540769619342 HOLD ffffff290854e7c0 2 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769631855 RELE ffffff290854e7c0 3 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769655857 HOLD ffffff290854e7c0 2 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44540769661693 RELE ffffff290854e7c0 3 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
44543242932420 DESTROY ffffff290854e7c0 
44543242943836 RELE ffffff290854e7c0 2 genunix`netstack_zone_destroy+0x67:genunix`zsd_apply_destroy+0x1d1:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xe6:genunix`zone_destroy+0xe0:genunix`zone+0x1e7:unix`sys_syscall+0x196::

At 44537871483917, the count drops to 1 – so the leak is clearly after that. Looking at these stacks first by HOLD:

$ cat OS-4810-leak.2.out | awk '{ if ($1 ~ /^[0-9]/) { printf("\n"); printf("%s ", $0) } else { printf("%s:", $1); } }' | grep ffffff290854e7c0 | sort -n | tail -95 | grep HOLD | cut -c40- | sort | uniq -c | sort -n
   1 genunix`netstack_find_by_zoneid+0x4c:genunix`netstack_find_by_cred+0x3f:ip`udp_do_open+0x68:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a::
   1 genunix`netstack_find_by_zoneid+0x4c:genunix`netstack_find_by_cred+0x3f:ip`udp_do_open+0x68:ip`udp_open+0xd8:ip`udp_openv6+0x27:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a::
   1 genunix`netstack_get_current+0x44:lx_brand`lx_fix_netstack+0x1d:lx_brand`lx_brandsys+0xc5f:genunix`brandsys+0x6d:unix`sys_syscall+0x196::
   1 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237::
   1 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_open+0xd8:ip`udp_openv6+0x27:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237::
   2 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
   2 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`rts_open+0x32:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
   2 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
   2 ip`ill_lookup_on_name+0x13d:ip`ipif_lookup_on_name+0xc7:ip`ip_sioctl_addif+0x31a:ip`ip_process_ioctl+0x28d:ip`ip_wput_nondata+0x1ba:ip`ip_helper_wput+0x23:unix`putnext+0x225:genunix`strdoioctl+0x2fd:genunix`strioctl+0x348:genunix`ldi_ioctl+0x6c:ip`udp_ioctl+0x64:sockfs`so_ioctl+0x1b9:sockfs`socket_ioctl+0x27:sockfs`socket_vop_ioctl+0x29:genunix`fop_ioctl+0x55:genunix`ioctl+0x9b:unix`_sys_sysenter_post_swapgs+0x153::
   2 ip`ipcl_conn_create+0x57:ip`rts_open+0x56:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
   2 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
   3 genunix`netstack_get_current+0x44:genunix`netstack_find_by_cred+0x4d:ip`udp_do_open+0x68:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
   3 ip`ipcl_conn_create+0x57:ip`udp_do_open+0x94:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
   4 genunix`netstack_find_by_stackid+0xa0:genunix`stropen+0x3e6:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
  10 genunix`netstack_find_by_stackid+0xa0:ip`ip_ne_queue_func+0x1e:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
  10 genunix`netstack_find_by_stackid+0xa0:ipnet`ipnet_nicevent_task+0x2b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::

And then by RELE:

$ cat OS-4810-leak.2.out | awk '{ if ($1 ~ /^[0-9]/) { printf("\n"); printf("%s ", $0) } else { printf("%s:", $1); } }' | grep ffffff290854e7c0 | sort -n | tail -95 | grep RELE | cut -c40- | sort | uniq -c | sort -n
   1 genunix`netstack_zone_destroy+0x67:genunix`zsd_apply_destroy+0x1d1:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xe6:genunix`zone_destroy+0xe0:genunix`zone+0x1e7:unix`sys_syscall+0x196::
   1 ip`udp_do_open+0xad:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
   1 ip`udp_do_open+0xad:ip`udp_open+0xd8:ip`udp_openv6+0x27:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
   2 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`ldi_open_by_vp+0xcc:genunix`ldi_open_by_name+0x87:ip`ip_interface_cleanup+0x91:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
   2 ip`ill_delete_tail+0x292:ip`ip_sioctl_removeif+0x31f:ip`ip_process_ioctl+0x102:ip`ip_wput_nondata+0x1ba:ip`udp_wput_iocdata+0x4c:ip`udp_wput_other+0x100:ip`udp_wput+0xd7:unix`put+0x1b5:ip`mi_copyin+0xed:ip`ip_sioctl_copyin_setup+0x305:ip`ip_wput_nondata+0x25b:ip`udp_wput_other+0x73:ip`udp_wput+0xd7:unix`putnext+0x225:genunix`strdoioctl+0x3d2:genunix`strioctl+0x1e5a:genunix`ldi_ioctl+0x6c:ip`ip_ioctl+0x44:ip`ip_loopback_removeif+0x78:ip`ip_interface_cleanup+0xda::
   2 ip`ipcl_conn_destroy+0x1bb:ip`rts_common_close+0x47:ip`rts_close+0x1e:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
   2 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_tpi_close+0x2c:genunix`qdetach+0x9b:genunix`strclose+0x2c1:specfs`device_close+0xb9:specfs`spec_close+0x17b:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
   2 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_tpi_close+0x2c:genunix`qdetach+0x9b:genunix`strclose+0x2c1:specfs`device_close+0xb9:specfs`spec_close+0x17b:genunix`fop_close+0x61:genunix`ldi_close+0x93:ip`ip_interface_cleanup+0xfa:ip`ip_stack_shutdown+0x2e:genunix`netstack_apply_shutdown+0xeb:genunix`apply_all_modules_reverse+0x49:genunix`netstack_zone_shutdown+0x10b:genunix`zsd_apply_shutdown+0x1c9:genunix`zsd_apply_all_keys+0x5f:genunix`zone_zsd_callbacks+0xd7:genunix`zone_shutdown+0x11a:genunix`zone+0x237:unix`sys_syscall+0x196::
   2 ip`rts_open+0x69:ip`rts_create+0x63:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
   2 ip`udp_do_open+0xad:ip`udp_open+0xd8:ip`udp_openv4+0x24:genunix`qattach+0xf6:genunix`stropen+0x347:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
   3 ip`ipcl_conn_destroy+0x1bb:ip`udp_do_close+0x67:ip`udp_close+0x19:sockfs`so_close+0x7a:sockfs`socket_close_internal+0x1b:sockfs`socket_vop_close+0xe8:genunix`fop_close+0x61:genunix`closef+0x5e:genunix`closeandsetf+0x398:genunix`close+0x13:unix`_sys_sysenter_post_swapgs+0x153::
   3 ip`udp_do_open+0xad:ip`udp_create+0x69:sockfs`socket_init_common+0x1c0:sockfs`so_init+0x25:sockfs`socket_create+0xc2:sockfs`so_socket+0x228:unix`_sys_sysenter_post_swapgs+0x153::
   4 genunix`stropen+0x407:specfs`spec_open+0x379:genunix`fop_open+0x89:genunix`vn_openat+0x234:genunix`copen+0x1fc:genunix`openat32+0x27:genunix`open32+0x25:unix`_sys_sysenter_post_swapgs+0x153::
  10 ip`ip_ne_queue_func+0x5b:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::
  10 ipnet`ipnet_nicevent_task+0x75:genunix`taskq_thread+0x2d0:unix`thread_start+0x8::

Looking through these shows a clear assymetry in that there is a HOLD from the following stack that has no corresponding RELE:

44537975533511 HOLD ffffff290854e7c0 3
              genunix`netstack_get_current+0x44
              lx_brand`lx_fix_netstack+0x1d
              lx_brand`lx_brandsys+0xc5f
              genunix`brandsys+0x6d
              unix`sys_syscall+0x196

Looking at the code, it is obvious why this is the case: lx_fix_netstack is missing a call to netstack_rele. This is the good news and it likely explains the rash of these panics since lx_fix_netstack integrated ~two months ago; the bad news is that this can't possibly explain the much older dumps. I will open a new ticket on lx_fix_netstack, and leave this ticket open to explore the older netstack reference count leak.

While we have not entirely root-caused all of the OS-4810 related failures (and indeed, because they are reference count leaks, getting to a root cause may be impossible), we are going to make changes to make the failure mode better fit the pathology: instead of implicitly tossing when reusing the netstack, we will not pick up a netstack in use.

#1

Updated by Electric Monk over 4 years ago

  • Status changed from New to Closed

git commit 854956ce2a18fd37e3f6160d38ffb87fdbc2edc4

commit  854956ce2a18fd37e3f6160d38ffb87fdbc2edc4
Author: Bryan Cantrill <bryan@joyent.com>
Date:   2016-10-11T18:14:30.000Z

    7413 netstack leaks lead to blind netstack reuse
    7414 iptun leaks netstacks
    7415 ipnet_promisc_add() leaks netstacks in error paths
    7416 ill_lookup_on_ifindex_global_instance() could leak netstack ids
    Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF