Project

General

Profile

Bug #3222

rcapd behaves poorly when under extreme load

Added by Piotr Jasiukajtis about 8 years ago. Updated about 8 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
cmd - userland programs
Start date:
2012-09-25
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

Joyent has a fix for that called:

OS-11 rcapd behaves poorly when under extreme load
[[https://github.com/joyent/illumos-joyent/commit/9230f57fd5dc4b0898a592a83cfa3f7b18b26718]]

Under a higher load when multiple zones are used 'rcapd' can consume almost 100% single cpu core at SYS.

CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0   47   881  538 3667   13  142  107    0  8444    2   8   0  90
  1    0   0   29  3106 1999 4212   10   94   71    0  8332    2  24   0  74
  2    0   0   28   169   48 3496   14   86   57    0  4091    1   6   0  93
  3    0   0   23   234   68 3105   14  103   60    0  7464    1  12   0  87
  4    0   0   47  2500 1356 1021   21   55   42    0  1546    0  93   0   7
  5    0   0  381  2712 1028 5456  176  420    7    0  5649    1  56   0  43
  6    0   0  359  3849 2112 5336  201  398   11    0  5804    1  58   0  41
  7    0   0    0  8083 4012    0    0    0    4    0     0    0 100   0   0

# dtrace -n 'profile-997hz /arg0 && execname == "rcapd"/ { @s[stack()] = count();} tick-15sec { trunc(@s, 7); printa(@s); exit(0); }'
dtrace: description 'profile-997hz ' matched 2 probes
CPU     ID                    FUNCTION:NAME
 14  77696                      :tick-15sec 

              unix`page_exists+0x77
              genunix`vmu_calculate_seg+0x3ac
              genunix`vmu_calculate_proc+0x259
              genunix`vmu_calculate+0xd2
              genunix`vm_getusage+0x23a
              genunix`rusagesys+0x64
              unix`sys_syscall+0x17a
              201

              unix`page_exists+0x7d
              genunix`vmu_calculate_seg+0x3ac
              genunix`vmu_calculate_proc+0x259
              genunix`vmu_calculate+0xd2
              genunix`vm_getusage+0x23a
              genunix`rusagesys+0x64
              unix`sys_syscall+0x17a
              205

              genunix`vmu_calculate_seg+0x38c
              genunix`vmu_calculate_proc+0x259
              genunix`vmu_calculate+0xd2
              genunix`vm_getusage+0x23a
              genunix`rusagesys+0x64
              unix`sys_syscall+0x17a
              223

              unix`page_exists+0x69
              genunix`vmu_calculate_seg+0x3ac
              genunix`vmu_calculate_proc+0x259
              genunix`vmu_calculate+0xd2
              genunix`vm_getusage+0x23a
              genunix`rusagesys+0x64
              unix`sys_syscall+0x17a
              940

              genunix`vmu_calculate_seg+0x3c2
              genunix`vmu_calculate_proc+0x259
              genunix`vmu_calculate+0xd2
              genunix`vm_getusage+0x23a
              genunix`rusagesys+0x64
              unix`sys_syscall+0x17a
             1193

              unix`page_exists+0x74
              genunix`vmu_calculate_seg+0x3ac
              genunix`vmu_calculate_proc+0x259
              genunix`vmu_calculate+0xd2
              genunix`vm_getusage+0x23a
              genunix`rusagesys+0x64
              unix`sys_syscall+0x17a
             3687

              unix`page_exists+0x79
              genunix`vmu_calculate_seg+0x3ac
              genunix`vmu_calculate_proc+0x259
              genunix`vmu_calculate+0xd2
              genunix`vm_getusage+0x23a
              genunix`rusagesys+0x64
              unix`sys_syscall+0x17a
             5057

As a workaround one may need to disable capped-memory from zones:

zonecfg -z zone1 remove capped-memory
zoneadm -z zone1 halt
zoneadm -z zone1 boot
#1

Updated by Rich Lowe about 8 years ago

  • Subject changed from OS-11 rcapd behaves poorly when under extreme load to rcapd behaves poorly when under extreme load

Also available in: Atom PDF