Project

General

Profile

Bug #8381

Convert ipsec_alg_lock from mutex to rwlock

Added by Dan McDonald over 3 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Low
Assignee:
Category:
networking
Start date:
2017-06-13
Due date:
% Done:

100%

Estimated time:
Difficulty:
Bite-size
Tags:
needs-triage
Gerrit CR:

Description

This is an early step of rewhacking some bits of IPsec. Some of the early attempted code included this particular bugfix, which is not a large performance win, but does keep exclusion out of IPsec algorithm instances except in cases of ipsecalgs(1M) instantiations.

Original work done by Bayard Bell.


Files

acqspray.sh (2.06 KB) acqspray.sh Dan McDonald, 2017-06-16 12:32 AM

History

#1

Updated by Dan McDonald over 3 years ago

Initial testing.

1.) Find address of global netstack's ipsec_alg_lock. Using mdb -k...

a.) ::netstack
b.) <global-netstack>::print netstack_t
c.) <ru_ipsec>::print -at ipsec_stack_t ipsec_alg_lock

Pre-this-fix it's a mutex, post-this-fix it's an rwlock.

Then use lockstat -A...

MUTEX VERSION:

# lockstat -A -l 0xffffff025c1aa598,8 ipsecalgs -s

Adaptive mutex hold: 3 events in 0.069 seconds (44 events/sec)

Count indv cuml rcnt     nsec Lock                   Caller                  
-------------------------------------------------------------------------------
    1  33%  33% 0.00   127641 0xffffff025c1aa598     spdsock_merge_algs+0x2e3
    1  33%  67% 0.00     4289 0xffffff025c1aa598     esp_register_out+0x2b4  
    1  33% 100% 0.00     3579 0xffffff025c1aa598     ah_register_out+0x1fe   
-------------------------------------------------------------------------------
# lockstat -A -l 0xffffff025c1aa598,8 ipsecalgs -l | tail -4
Count indv cuml rcnt     nsec Lock                   Caller                  
-------------------------------------------------------------------------------
    1 100% 100% 0.00    22319 0xffffff025c1aa598     spdsock_dumpalgs+0x34a  
-------------------------------------------------------------------------------
# 

RWLOCK VERSION:

# lockstat -A -l 0xffffff025c01d598,8 ipsecalgs -s

R/W writer hold: 1 events in 0.069 seconds (14 events/sec)

Count indv cuml rcnt     nsec Lock                   Caller                  
-------------------------------------------------------------------------------
    1 100% 100% 0.00   125397 0xffffff025c01d598     spdsock_merge_algs+0x2e3
-------------------------------------------------------------------------------

R/W reader hold: 2 events in 0.069 seconds (29 events/sec)

Count indv cuml rcnt     nsec Lock                   Caller                  
-------------------------------------------------------------------------------
    1  50%  50% 0.00     5116 0xffffff025c01d598     ah_register_out+0x1fe   
    1  50% 100% 0.00     4496 0xffffff025c01d598     esp_register_out+0x2bc  
-------------------------------------------------------------------------------
# 

Count indv cuml rcnt     nsec Lock                   Caller                  
-------------------------------------------------------------------------------
    1 100% 100% 0.00    22990 0xffffff025c01d598     spdsock_dumpalgs+0x352  
-------------------------------------------------------------------------------
#2

Updated by Dan McDonald over 3 years ago

Further testing provided by the acqspray.sh attached here. (A variant of which may get upstreamed to usr/src/test later.)

Sample output:

MUTEX VERSION:

# ./acqspray.sh
WARNING -- this test flushes out IPsec policy...
GLOBAL_NETSTACK = ffffff02545b2dc0
GLOBAL_IPSEC = 0xffffff025b08b000
IPSEC_ALG_LOCK = ffffff025b08b598

        IPsec policy should be managed using smf(5). Modifying
        the IPsec policy from the command line while the 'policy'
        service is enabled could result in an inconsistent
        security policy.

        IPsec policy should be managed using smf(5). Modifying
        the IPsec policy from the command line while the 'policy'
        service is enabled could result in an inconsistent
        security policy.

        WARNING : New policy entries that are being added may
        affect the existing connections. Existing connections
        that are not subjected to policy constraints, may be
        subjected to policy constraints because of the new
        policy. This can disrupt the communication of the
        existing connections.

#INDEX 13
{ raddr 10.21.12.0/24 dir out } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared } 
#INDEX 14
{ raddr 10.21.12.0/24 dir in } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared } 
Launching in.iked temporarily, using /dev/null as config file.
Running:     lockstat -A -l 0xffffff025b08b598,8 sleep 30

Adaptive mutex hold: 765 events in 30.077 seconds (25 events/sec)

Count indv cuml rcnt     nsec Lock                   Caller                  
-------------------------------------------------------------------------------
  510  67%  67% 0.00     1672 0xffffff025b08b598     sadb_new_algdesc+0x134  
  255  33% 100% 0.00    26694 0xffffff025b08b598     esp_send_acquire+0xfb   
-------------------------------------------------------------------------------
# 

RWLOCK VERSION:

# ./acqspray.sh
WARNING -- this test flushes out IPsec policy...
GLOBAL_NETSTACK = ffffff02545b2dc0
GLOBAL_IPSEC = 0xffffff025be4b000
IPSEC_ALG_LOCK = ffffff025be4b598

        IPsec policy should be managed using smf(5). Modifying
        the IPsec policy from the command line while the 'policy'
        service is enabled could result in an inconsistent
        security policy.

        IPsec policy should be managed using smf(5). Modifying
        the IPsec policy from the command line while the 'policy'
        service is enabled could result in an inconsistent
        security policy.

        WARNING : New policy entries that are being added may
        affect the existing connections. Existing connections
        that are not subjected to policy constraints, may be
        subjected to policy constraints because of the new
        policy. This can disrupt the communication of the
        existing connections.

#INDEX 7
{ raddr 10.21.12.0/24 dir out } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared } 
#INDEX 8
{ raddr 10.21.12.0/24 dir in } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared } 
Launching in.iked temporarily, using /dev/null as config file.
Running:     lockstat -A -l 0xffffff025be4b598,8 sleep 30

R/W reader hold: 765 events in 30.085 seconds (25 events/sec)

Count indv cuml rcnt     nsec Lock                   Caller                  
-------------------------------------------------------------------------------
  510  67%  67% 0.00     1611 0xffffff025be4b598     sadb_new_algdesc+0x123  
  255  33% 100% 0.00    24576 0xffffff025be4b598     esp_send_acquire+0xfb   
-------------------------------------------------------------------------------
# 
#3

Updated by Electric Monk over 3 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 69e7133115efafd5527a1c8b7735b0e8937633f6

commit  69e7133115efafd5527a1c8b7735b0e8937633f6
Author: Bayard Bell <buffer.g.overflow@gmail.com>
Date:   2017-06-17T02:42:11.000Z

    8381 Convert ipsec_alg_lock from mutex to rwlock
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Reviewed by: Gordon Ross <gordon.w.ross@gmail.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom PDF