Bug #8560

Reference leak on ipsec_action_t

Added by Dan McDonald 12 months ago. Updated 6 months ago.

Status:ClosedStart date:2017-08-04
Priority:LowDue date:
Assignee:-% Done:

100%

Category:networking
Target version:-
Difficulty:Medium Tags:needs-triage

Description

While verifying #8554, I found one last leaked object. It has a reference count of 1214.

> ffffff0da4a718a8::print ipsec_action_t
{
    ipa_hash = {
        hash_next = 0
        hash_pp = 0xffffff0db9ed1018
    }
    ipa_next = 0
    ipa_refs = 0x4be
    ipa_act = {
        ipa_type = 0x1
        ipa_log = 0
        ipa_u = {
            ipau_apply = {
                ipp_use_ah = 0
                ipp_use_esp = 0x1
                ipp_use_se = 0
                ipp_use_unique = 0
                ipp_use_espa = 0x1
                ipp_pad = 0
                ipp_auth_alg = 0
                ipp_encr_alg = 0xc
                ipp_esp_auth_alg = 0x7
                ipp_ah_minbits = 0
                ipp_ah_maxbits = 0
                ipp_espe_minbits = 0x80
                ipp_espe_maxbits = 0x100
                ipp_espa_minbits = 0x200
                ipp_espa_maxbits = 0x200
                ipp_km_proto = 0
                ipp_km_cookie = 0
                ipp_replay_depth = 0
            }
            ipau_reject_type = 0x12
            ipau_resolve_id = 0x12
            ipau_log_type = 0x12
        }
    }
    ipa_hval = 0xfe
    ipa_allow_clear = 0
    ipa_want_ah = 0
    ipa_want_esp = 0x1
    ipa_want_se = 0
    ipa_want_unique = 0
    ipa_pad = 0x5d6ee
    ipa_ovhd = 0x4e
}
> 

It's one action, and it appears to be (like #8554) only an issue on IPsec-using netstacks that shut down.

History

#1 Updated by Dan McDonald 7 months ago

Reproduction:

1.) Set up a non-global zone with IPsec. It can communicate with the global zone if you only have one machine.

2.) Communicate to this NGZ using IPsec. More traffic is better. Pings, TCP, whatever.

3.) Preferably while traffic is moving, utter "halt" or "poweroff" from inside the zone. Using zoneadm(1M) could work too, but I've had better results with in-zone halting.

4.) Wait until the zone is fully halted. If you see this printout on /var/adm/message or console:

3288 (void) printf("ipsec_action_free_table(%p) ref %d\n",
3289 (void *)ap, ap->ipa_refs);

you're seeing the leak, which is detectable by ::findleaks in a kernel dump.

#2 Updated by Dan McDonald 6 months ago

Proposed fix is to make sure ip_recv_attr_t references to actions actually release their reference:

diff --git a/usr/src/uts/common/inet/ip/spd.c b/usr/src/uts/common/inet/ip/spd.c
index 841c345..5dedfab 100644
--- a/usr/src/uts/common/inet/ip/spd.c
+++ b/usr/src/uts/common/inet/ip/spd.c
@@ -4123,6 +4123,11 @@ ipsec_in_release_refs(ip_recv_attr_t *ira)
                IPSA_REFRELE(ira->ira_ipsec_esp_sa);
                ira->ira_ipsec_esp_sa = NULL;
        }
+       if (ira->ira_ipsec_action != NULL) {
+               IPACT_REFRELE(ira->ira_ipsec_action);
+               ira->ira_ipsec_action = NULL;
+       }
+
        ira->ira_flags &= ~IRAF_IPSEC_SECURE;
 }

Synthetic testing using the above reproduction showed no leaks (and no /var/adm/messages messages) with the fix. Testing on a multi-zone OmniOS server using IPsec in one zone (which was halted-and-restarted at least twice during the before-and-after test runs) also shows no leaks.

Before the fix:

neuromancer(~/dump/8560)[0]% mdb 0
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci zfs sata sd ip hook neti sockfs arp usba xhci stmf stmf_sbd mm lofs random idm cpc kvm crypto ufs logindmux ptm nsmb smbsrv nfs ]
> ::findleaks
CACHE             LEAKED           BUFCTL CALLER
ffffff0d5d686448       1 ffffff0d81bcc3a8 ipsec_act_find+0x141
------------------------------------------------------------------------
           Total       1 buffer, 72 bytes
> ::zone
            ADDR     ID STATUS        NAME                 PATH
fffffffffc02e5e0      0 running       global               /
ffffff0d8fc23100      1 running       XXX       
/zones/XXX/root/
ffffff0d8fc1c200      2 running       webserver            
/zones/webserver/root/
ffffff0d8fc1a800      3 running       omniti               /zones/omniti/root/
ffffff0d98ec2a80      5 running       YYY            
/zones/YYY/root/
ffffff0d80b52b00      8 running       router               /zones/router/root/
> 

After the fix:

neuromancer(~/dump/8560)[0]% mdb 1
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci zfs sata sd ip hook neti sockfs arp usba xhci stmf stmf_sbd mm lofs random idm cpc ufs logindmux ptm smbsrv nfs ]
> ::findleaks
findleaks: no memory leaks detected
> ::zone 
            ADDR     ID STATUS        NAME                 PATH
fffffffffc02e5e0      0 running       global               /
ffffff0d7f80a140      1 running       XXX       
/zones/XXX/root/
ffffff0d7f813080      2 running       YYY            
/zones/minecraft/root/
ffffff0d87d91700      4 running       webserver            
/zones/webserver/root/
ffffff0d87d90a00      5 running       omniti               /zones/omniti/root/
ffffff11c5dd8e80      7 running       router               /zones/router/root/
> 

#3 Updated by Electric Monk 6 months ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

git commit 620632424515d984b96f0a398e2d0cba9412182c

commit  620632424515d984b96f0a398e2d0cba9412182c
Author: Dan McDonald <danmcd@joyent.com>
Date:   2018-01-18T16:25:00.000Z

    8560 Reference leak on ipsec_action_t
    Reviewed by: Norm Jacobs <naj@snapcon.com>
    Reviewed by: Andy Fiddaman <omnios@citrus-it.net>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom