illumos gate

Please make the vmdump available? If there's sensitive information on it, I can assure you (esp. given it's a SmartOS dump) we can respect the privacy of the dump in question.

Got the coredump. Some initial observations:

> $C
fffffbe33059ab10 udp_do_bind+0x76(fffffdfa95564580, fffffbe33059ab28, 20, fffffdfca739fc88, 0)
fffffbe33059ab60 udp_implicit_bind+0x61(fffffdfa95564580, fffffdfca739fc88)
fffffbe33059ac00 udp_send+0x1a8(fffffdfa95564580, fffffdf586b34520, fffffbe33059ad18, fffffdfca739fc88)
fffffbe33059ac90 so_sendmsg+0x252(fffffdfde0f502d8, fffffbe33059ad18, fffffbe33059ae78, fffffdfca739fc88)
fffffbe33059acf0 socket_sendmsg+0x62(fffffdfde0f502d8, fffffbe33059ad18, fffffbe33059ae78, fffffdfca739fc88)
fffffbe33059ad90 socket_vop_write+0x61(fffffdfde0f4fd80, fffffbe33059ae78, 0, fffffdfca739fc88, 0)
fffffbe33059ae30 fop_write+0x111(fffffdfde0f4fd80, fffffbe33059ae78, 0, fffffdfca739fc88, 0)
fffffbe33059af00 write+0x2c6(14b, c02e326700, 350)
fffffbe33059af10 sys_syscall+0x1a8()

So it died in udp_do_bind() as part of the implicit-bind one does with a sendto() or sendmsg() (or write() in this case?).

A quick look at the code suggests the conn_t has a NULL netstack for it to panic there, and sure enough:

> fffffdfa95564580::print -at conn_t conn_netstack
fffffdfa95564740 netstack_t *conn_netstack = 0
> 

So what's interesting about this bug & dump is: How did a conn_t end up with a NULL netstack pointer? To be continued...

One more piece of data, this appears to be use-after-free:

> fffffdfa95564580::whatis
fffffdfa95564580 is freed from udp_conn_cache

So now to figure out what two threads running together can cause this.

The bug filer, running a DEBUG kernel in his environment, has exposed a race between socket closing and the new-ish #9531 implementation. Basically, for TCP sockets, the "so_closed" upcall destroys state that the #9531 code accesses. This webrev: https://kebe.com/~danmcd/webrevs/12976/ narrows or eliminates the race by making the set-upcall-data-to-NULL portion protected by conn_lock, and the #9511 code call the upcall while under conn_lock as well.

It has been tested by the filer with DEBUG without a panic. This coming week, the filer will get a non-DEBUG kernel with this fix.

Jason now has non-DEBUG production time up with this fix too, and he is happy.

Note that the #9531 implementation is tickled by netstat(1) processes. Additional testing has demonstrated that `netstat -u` continues to function as before.