illumos gate

In production we were seeing several bad pkt state alloc kstats beings bumped from ipf on several of our zones.

Running the following D script in us-east-*:

#define FI_SHORT    0x0008
#define FI_STATE    0x0100
#define FI_FRAGBODY 0x2000
#define FI_BAD      0x0400
#define FI_OOW 0x0800
#define TH_SYN 0x02
#define IPPROTO_ICMP 1
#define IPPROTO_ICMPV6 58
#define IPPROTO_GRE 47
#define IPPROTO_TCP 6
#define IPPROTO_UDP 17

struct icmp {
    uchar_t icmp_type;      /* type of message, see below */
    uchar_t icmp_code;      /* type sub code */
    /* ... */
};

fr_addstate:entry { self->fin = args[0]; }

fr_addstate:return
/args[1] == NULL/
{
        this->ifs = self->fin->fin_ifs;
        this->fin_flx = self->fin->fin_fi.fi_flx;
        this->proto = self->fin->fin_fi.fi_p;
        printf("[%s] proto=%s %d:%d%s, fin->fin_flx=%0x, ifs[ifs_fr_state_lock=%d, ifs_ips_num=%d, ifs_fr_statemax=%d], A=%d, B=%d, fr[statemax=%d]\\n",
                zonename,                (this->proto == IPPROTO_ICMP ? "icmp:" :
                (this->proto == IPPROTO_ICMPV6 ? "icmpv6" :
                (this->proto == IPPROTO_GRE ? "gre" :                (this->proto == IPPROTO_TCP ? "tcp" :
                (this->proto == IPPROTO_UDP ? "udp" :
                "?"                ))))),
                this->proto == IPPROTO_ICMP ? ((struct icmp *)self->fin->fin_dp)->icmp_type : 0,
                this->proto == IPPROTO_ICMP ? ((struct icmp *)self->fin->fin_dp)->icmp_code : 0,                args[1] == NULL ? " - NULL" : "",
                this->fin_flx,
                this->ifs->ifs_fr_state_lock,
                this->ifs->ifs_ips_num,
                this->ifs->ifs_fr_statemax,
                (this->fin_flx & (FI_SHORT|FI_STATE|FI_FRAGBODY|FI_BAD)),
                ((this->fin_flx & FI_OOW) && !(self->fin->fin_tcpf & TH_SYN)),
                self->fin->fin_fr ? self->fin->fin_fr->fr_statemax : 44
        );
}

This reveals that the packets causing the kstats to be bumped are ICMP replies (code 0, type 0), causing output like:

  4   2512               fr_addstate:return [global] proto=icmp: 0:0, fin->fin_flx=0, ifs[ifs_fr_state_lock=0, ifs_ips_num=703, ifs_fr_statemax=113279], A=0, B=0, fr[statemax=0]

Looking at fr_addstate(), we can see that ICMP_ECHOREPLY is not covered, and so the function returns NULL: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/inet/ipf/ip_state.c#1282

The good news here is that this isn't affecting the data flow for manta, but it is preventing the LB zones from being pingable.