Bug #8381
Convert ipsec_alg_lock from mutex to rwlock
Start date:
2017-06-13
Due date:
% Done:
100%
Estimated time:
Difficulty:
Bite-size
Tags:
needs-triage
Gerrit CR:
Description
This is an early step of rewhacking some bits of IPsec. Some of the early attempted code included this particular bugfix, which is not a large performance win, but does keep exclusion out of IPsec algorithm instances except in cases of ipsecalgs(1M) instantiations.
Original work done by Bayard Bell.
Files
Updated by Dan McDonald over 3 years ago
Initial testing.
1.) Find address of global netstack's ipsec_alg_lock. Using mdb -k...
a.) ::netstack
b.) <global-netstack>::print netstack_t
c.) <ru_ipsec>::print -at ipsec_stack_t ipsec_alg_lock
Pre-this-fix it's a mutex, post-this-fix it's an rwlock.
Then use lockstat -A...
MUTEX VERSION:
# lockstat -A -l 0xffffff025c1aa598,8 ipsecalgs -s
Adaptive mutex hold: 3 events in 0.069 seconds (44 events/sec)
Count indv cuml rcnt nsec Lock Caller
-------------------------------------------------------------------------------
1 33% 33% 0.00 127641 0xffffff025c1aa598 spdsock_merge_algs+0x2e3
1 33% 67% 0.00 4289 0xffffff025c1aa598 esp_register_out+0x2b4
1 33% 100% 0.00 3579 0xffffff025c1aa598 ah_register_out+0x1fe
-------------------------------------------------------------------------------
# lockstat -A -l 0xffffff025c1aa598,8 ipsecalgs -l | tail -4
Count indv cuml rcnt nsec Lock Caller
-------------------------------------------------------------------------------
1 100% 100% 0.00 22319 0xffffff025c1aa598 spdsock_dumpalgs+0x34a
-------------------------------------------------------------------------------
#
RWLOCK VERSION:
# lockstat -A -l 0xffffff025c01d598,8 ipsecalgs -s
R/W writer hold: 1 events in 0.069 seconds (14 events/sec)
Count indv cuml rcnt nsec Lock Caller
-------------------------------------------------------------------------------
1 100% 100% 0.00 125397 0xffffff025c01d598 spdsock_merge_algs+0x2e3
-------------------------------------------------------------------------------
R/W reader hold: 2 events in 0.069 seconds (29 events/sec)
Count indv cuml rcnt nsec Lock Caller
-------------------------------------------------------------------------------
1 50% 50% 0.00 5116 0xffffff025c01d598 ah_register_out+0x1fe
1 50% 100% 0.00 4496 0xffffff025c01d598 esp_register_out+0x2bc
-------------------------------------------------------------------------------
#
Count indv cuml rcnt nsec Lock Caller
-------------------------------------------------------------------------------
1 100% 100% 0.00 22990 0xffffff025c01d598 spdsock_dumpalgs+0x352
-------------------------------------------------------------------------------
Updated by Dan McDonald over 3 years ago
- File acqspray.sh acqspray.sh added
Further testing provided by the acqspray.sh attached here. (A variant of which may get upstreamed to usr/src/test later.)
Sample output:
MUTEX VERSION:
# ./acqspray.sh
WARNING -- this test flushes out IPsec policy...
GLOBAL_NETSTACK = ffffff02545b2dc0
GLOBAL_IPSEC = 0xffffff025b08b000
IPSEC_ALG_LOCK = ffffff025b08b598
IPsec policy should be managed using smf(5). Modifying
the IPsec policy from the command line while the 'policy'
service is enabled could result in an inconsistent
security policy.
IPsec policy should be managed using smf(5). Modifying
the IPsec policy from the command line while the 'policy'
service is enabled could result in an inconsistent
security policy.
WARNING : New policy entries that are being added may
affect the existing connections. Existing connections
that are not subjected to policy constraints, may be
subjected to policy constraints because of the new
policy. This can disrupt the communication of the
existing connections.
#INDEX 13
{ raddr 10.21.12.0/24 dir out } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared }
#INDEX 14
{ raddr 10.21.12.0/24 dir in } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared }
Launching in.iked temporarily, using /dev/null as config file.
Running: lockstat -A -l 0xffffff025b08b598,8 sleep 30
Adaptive mutex hold: 765 events in 30.077 seconds (25 events/sec)
Count indv cuml rcnt nsec Lock Caller
-------------------------------------------------------------------------------
510 67% 67% 0.00 1672 0xffffff025b08b598 sadb_new_algdesc+0x134
255 33% 100% 0.00 26694 0xffffff025b08b598 esp_send_acquire+0xfb
-------------------------------------------------------------------------------
#
RWLOCK VERSION:
# ./acqspray.sh
WARNING -- this test flushes out IPsec policy...
GLOBAL_NETSTACK = ffffff02545b2dc0
GLOBAL_IPSEC = 0xffffff025be4b000
IPSEC_ALG_LOCK = ffffff025be4b598
IPsec policy should be managed using smf(5). Modifying
the IPsec policy from the command line while the 'policy'
service is enabled could result in an inconsistent
security policy.
IPsec policy should be managed using smf(5). Modifying
the IPsec policy from the command line while the 'policy'
service is enabled could result in an inconsistent
security policy.
WARNING : New policy entries that are being added may
affect the existing connections. Existing connections
that are not subjected to policy constraints, may be
subjected to policy constraints because of the new
policy. This can disrupt the communication of the
existing connections.
#INDEX 7
{ raddr 10.21.12.0/24 dir out } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared }
#INDEX 8
{ raddr 10.21.12.0/24 dir in } ipsec { encr_algs aes-cbc(128..256) encr_auth_algs hmac-sha512(512) sa shared }
Launching in.iked temporarily, using /dev/null as config file.
Running: lockstat -A -l 0xffffff025be4b598,8 sleep 30
R/W reader hold: 765 events in 30.085 seconds (25 events/sec)
Count indv cuml rcnt nsec Lock Caller
-------------------------------------------------------------------------------
510 67% 67% 0.00 1611 0xffffff025be4b598 sadb_new_algdesc+0x123
255 33% 100% 0.00 24576 0xffffff025be4b598 esp_send_acquire+0xfb
-------------------------------------------------------------------------------
#
Updated by Electric Monk over 3 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit 69e7133115efafd5527a1c8b7735b0e8937633f6
commit 69e7133115efafd5527a1c8b7735b0e8937633f6
Author: Bayard Bell <buffer.g.overflow@gmail.com>
Date: 2017-06-17T02:42:11.000Z
8381 Convert ipsec_alg_lock from mutex to rwlock
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Gordon Ross <gordon.w.ross@gmail.com>
Approved by: Richard Lowe <richlowe@richlowe.net>