illumos gate

Joyent saw a panic in the field. Here is our analysis:

> ::status
debugging crash dump vmcore.1 (64-bit) from xxxxxx
operating system: 5.11 joyent_20110922T212927Z (i86pc)
image uuid: (not set)
panic message: BAD TRAP: type=e (#pf Page fault) rp=ffffff00b8fb3750
addr=28 occurred in module "ip" due to a NULL pointer dereference
dump content: kernel pages only
> $C
ffffff00b8fb38b0 conn_ip_output+0x4b(ffffff1a38f73de0, ffffff19f0706b00)
ffffff00b8fb3930 tcp_ss_rexmit+0x191(ffffff19f07053c0)
ffffff00b8fb3950 tcp_rexmit_after_error+0x82(ffffff19f07053c0)
ffffff00b8fb39a0 tcp_icmp_input+0x1d2(ffffff19f07050c0,
ffffff1a38f568a0, ffffff199f447c80, ffffff00b8fb39f0)
ffffff00b8fb3bc0 squeue_drain+0x1f8(ffffff199f447c80, 2, 129ccb0ec34dc)
ffffff00b8fb3c20 squeue_worker+0x132(ffffff199f447c80)
ffffff00b8fb3c30 thread_start+8()
> ffffff19f07053c0::print tcp_t tcp_state
tcp_state = 0xfffffffa
>::print tcp_t tcp_state | =D
                -6

State -6 corresponds to a closed TCP connection.

          
> ffffff19f07050c0::print conn_t conn_state_flags
conn_state_flags = 0x3

This conn_state corresponds to CONN_CLOSED and CONN_CONDEMNED.

From here we basically can see that we had already closed our TCP connection. However, despite that we were trying to send on a TCP packet. The fix here is that tcp_icmp_input() should check whether or not the TCP connection is closed and if so, we should drop the packet.