illumos gate

Digging in to the source... the call itself:

 6990 mblk_t *
 6991 ip_input_fragment(mblk_t *mp, ipha_t *ipha, ip_recv_attr_t *ira)
...
 7014         ill_t           *ill = ira->ira_ill;
...
 7129         ipfb = &ill->ill_frag_hash_tbl[ILL_FRAG_HASH(src, ident)];
 7130         mutex_enter(&ipfb->ipfb_lock);

Find ill, which we believe to be ffffff00b8d933b0, which would be up in the
stack frame for the call to ip_input/ip_input_common_v4 -- this appears to make sense.

> ffffff00b8d933b0::print ip_recv_attr_t ira_ill
{
    ill_inputfn = ill_input_short_v4
...
    ill_name = 0xffffff19b4282be0 "lo0" 
...
    ill_frag_hash_tbl = 0

Looking back at the source:

 105 #define ILL_FRAG_HASH_TBL_COUNT ((unsigned int)64)
 106 #define ILL_FRAG_HASH_TBL_SIZE  (ILL_FRAG_HASH_TBL_COUNT * sizeof (ipfb_t))
...
2748 #define ILL_FRAG_HASH(s, i) \\
2749         ((ntohl(s) ^ ((i) ^ ((i) >> 8))) % ILL_FRAG_HASH_TBL_COUNT)

So, we need src and ident:

> 0xffffff19bd255014::print -ta ipha_t
  4 ffffff19bd255018 uint16_t ipha_ident = 0xe60a
  c ffffff19bd255020 ipaddr_t ipha_src = 2.231.19.148
...
src = 0x9413e702 --ntohl--> 0x02e71394
ident = 0xe60a

...
> ::sizeof ipfb_t
sizeof (ipfb_t) = 0x20

... (python)
>>> hex((0x02e71394 ^ ((0xe60a) ^ ((0xe60a) >> 8))) % 64)
'0x38'

> 0x38 * 0x20 = X
                700

And there's our crazy offset from the panic.

Digging further into the source, it seems like we generally only set (or unset) ill_frag_hash_tbl in one allocation (and free) path:

Assignments to: ill_frag_hash_tbl

  File    Function        Line
1 ip_if.c ill_delete_tail  609 ill->ill_frag_hash_tbl = NULL;
2 ip_if.c ill_init        3379 ill->ill_frag_hash_tbl = (ipfb_t *)frag_ptr;

The usual call graph for allocation looks something like:

ip_openv4()
  --> ip_open(... MODOPEN ...)
      --> ip_modopen()
          --> mi_open_alloc_sleep(sizeof (ill_t))
          --> ill_init()
              --> assign ill_frag_hash_tbl

At which point ill_frag_hash_tbl is populated and all is well.

There is special case code, though, for the lo0 loopback adapter. These appear to be created with some oddball properties in response to a lookup for same, in ill_lookup_on_name if called with do_alloc B_TRUE.

As it turns out, we only call with do_alloc B_TRUE via:

ip_process_ioctl(SIOCLIFADDIF)
  --> ip_sioctl_addif() /* for interface matching loopback name, i.e. lo* */
      --> ipif_lookup_on_name()

The panic is the combination of:

• ifconfig lo0 addif X.X.X.X up, in order to make a public IP address known as one of a zone's identities, but without putting that IP on a real interface
• a packet large enough to be fragmented is routed to the zone, say via a NPAT or DSR load balancing setup

This is generally a load balanced public IP setup where the actual public IP address is mastered on a network device which then routes (without changing source or destination IP addresses) the packets to the zone. So that the IP stack in the zone does not immediately discard this otherwise foreign destination, it is brought up on the loopback adapter, where applications in the zone can bind to it.

This occurs because ire_recv_local_v4 notices that the ingress ILL (e.g. net0) for the received packet is not the same as the ILL that hosts the target IP address (lo0). In this event, we use ip_check_multihome to swap in the target ILL (assuming such a thing is allowed) before calling into ip_input_local_v4.

For fragmented packets, we then call into ip_input_fragment for processing. This function expects that "each ILL has a hash table for queuing packets undergoing reassembly for all IPIFs associated with the ILL." As noted in earlier comments, we most definitely do not allocate this hash table for lo0 as allocated inline with a ill_lookup_on_name() call.

The fix, here, is to allocate a hash table for the new lo0 ill_t in ill_lookup_on_name() by rejigging that function to use the appropriate parts of ill_init().