Actions
Bug #5199
closed"bad pkt state alloc" kstats due to echo replies
Start date:
2014-10-01
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
Description
In production we were seeing several bad pkt state alloc kstats beings bumped from ipf on several of our zones.
Running the following D script in us-east-*:
#define FI_SHORT 0x0008 #define FI_STATE 0x0100 #define FI_FRAGBODY 0x2000 #define FI_BAD 0x0400 #define FI_OOW 0x0800 #define TH_SYN 0x02 #define IPPROTO_ICMP 1 #define IPPROTO_ICMPV6 58 #define IPPROTO_GRE 47 #define IPPROTO_TCP 6 #define IPPROTO_UDP 17 struct icmp { uchar_t icmp_type; /* type of message, see below */ uchar_t icmp_code; /* type sub code */ /* ... */ }; fr_addstate:entry { self->fin = args[0]; } fr_addstate:return /args[1] == NULL/ { this->ifs = self->fin->fin_ifs; this->fin_flx = self->fin->fin_fi.fi_flx; this->proto = self->fin->fin_fi.fi_p; printf("[%s] proto=%s %d:%d%s, fin->fin_flx=%0x, ifs[ifs_fr_state_lock=%d, ifs_ips_num=%d, ifs_fr_statemax=%d], A=%d, B=%d, fr[statemax=%d]\\n", zonename, (this->proto == IPPROTO_ICMP ? "icmp:" : (this->proto == IPPROTO_ICMPV6 ? "icmpv6" : (this->proto == IPPROTO_GRE ? "gre" : (this->proto == IPPROTO_TCP ? "tcp" : (this->proto == IPPROTO_UDP ? "udp" : "?" ))))), this->proto == IPPROTO_ICMP ? ((struct icmp *)self->fin->fin_dp)->icmp_type : 0, this->proto == IPPROTO_ICMP ? ((struct icmp *)self->fin->fin_dp)->icmp_code : 0, args[1] == NULL ? " - NULL" : "", this->fin_flx, this->ifs->ifs_fr_state_lock, this->ifs->ifs_ips_num, this->ifs->ifs_fr_statemax, (this->fin_flx & (FI_SHORT|FI_STATE|FI_FRAGBODY|FI_BAD)), ((this->fin_flx & FI_OOW) && !(self->fin->fin_tcpf & TH_SYN)), self->fin->fin_fr ? self->fin->fin_fr->fr_statemax : 44 ); }
This reveals that the packets causing the kstats to be bumped are ICMP replies (code 0, type 0), causing output like:
4 2512 fr_addstate:return [global] proto=icmp: 0:0, fin->fin_flx=0, ifs[ifs_fr_state_lock=0, ifs_ips_num=703, ifs_fr_statemax=113279], A=0, B=0, fr[statemax=0]
Looking at fr_addstate(), we can see that ICMP_ECHOREPLY is not covered, and so the function returns NULL: http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/inet/ipf/ip_state.c#1282
The good news here is that this isn't affecting the data flow for manta, but it is preventing the LB zones from being pingable.
Updated by Electric Monk over 7 years ago
- Status changed from New to Closed
git commit 0b9f216886fedfeee1db590ff8db3baf72dc9118
commit 0b9f216886fedfeee1db590ff8db3baf72dc9118 Author: Rob Gulewich <robert.gulewich@joyent.com> Date: 2014-12-11T02:00:28.000Z 5199 "bad pkt state alloc" kstats due to echo replies Reviewed by: Dan McDonald <danmcd@omniti.com> Reviewed by: Darren Reed <darrenr@fastmail.net> Approved by: Richard Lowe <richlowe@richlowe.net>
Actions