root@lab10:~# date September 6, 2011 07:21:51 AM PDT root@lab10:~# tail -4 /etc/system * August 30, 2011 02:26:39 PM PDT * try to solve CPU creep issue * https://defect.opensolaris.org/bz/show_bug.cgi?id=12799 set cpupm_tw_gov_interval=800000 ######################################################## # Comment - the above 'cpupm_tw_gov_interval' value # doesn't seem to stop the problem, but it might slow # the progresssion down ######################################################## root@lab10:~# rup lab10 lab10 up 6 days, 15:41, load average: 0.00, 0.00, 0.00 root@lab10:~# uname -a SunOS lab10 5.11 oi_148b i86pc i386 i86pc Solaris root@lab10:~# prstat -s cpu -n 5 > /tmp/prstat.s.cpu.n.5.out ^Croot@lab10:~# cat /tmp/prstat.s.cpu.n.5.out PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 1204 root 59M 36M sleep 59 0 0:31:48 0.0% Xorg/3 1657 gdm 87M 25M sleep 59 0 0:03:12 0.0% gdm-simple-gree/1 10 root 16M 13M sleep 59 0 0:00:02 0.0% svc.startd/16 894 root 6968K 5852K sleep 59 0 0:00:15 0.0% intrd/1 1 root 2728K 1860K sleep 59 0 0:00:00 0.0% init/1 Total: 65 processes, 393 lwps, load averages: 0.00, 0.00, 0.00 PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 1204 root 59M 36M sleep 59 0 0:31:48 0.1% Xorg/3 1657 gdm 87M 25M sleep 59 0 0:03:12 0.0% gdm-simple-gree/1 10 root 16M 13M sleep 59 0 0:00:02 0.0% svc.startd/16 894 root 6968K 5852K sleep 59 0 0:00:15 0.0% intrd/1 1 root 2728K 1860K sleep 59 0 0:00:00 0.0% init/1 Total: 65 processes, 393 lwps, load averages: 0.00, 0.00, 0.00 root@lab10:~# mpstat 5 2 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 1 514 101 90 0 1 0 0 9 0 13 0 87 1 0 0 0 81 9 38 0 1 0 0 7 0 2 0 98 2 0 0 0 73 8 35 0 1 0 0 6 0 2 0 98 3 0 0 0 67 7 32 0 1 0 0 5 0 2 0 98 4 0 0 0 75 8 37 0 1 0 0 6 0 2 0 98 5 0 0 0 64 6 31 0 1 0 0 6 0 2 0 98 6 0 0 0 65 6 31 0 1 0 0 6 0 2 0 98 7 0 0 0 65 6 31 0 1 0 0 5 0 2 0 98 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 513 101 128 0 1 1 0 36 0 26 0 73 1 0 0 0 56 2 29 0 0 0 0 6 0 3 0 97 2 0 0 0 34 3 15 0 0 0 0 6 0 2 0 98 3 0 0 0 80 6 35 0 0 1 0 0 0 4 0 96 4 5 0 0 59 1 30 0 1 1 0 61 0 3 0 97 5 0 0 0 72 4 33 0 1 0 0 0 0 4 0 96 6 0 0 0 82 14 40 0 0 0 0 0 0 5 0 95 7 0 0 0 32 3 16 0 0 0 0 12 0 2 0 98 root@lab10:~# root@lab10:~# powertop OpenIndiana PowerTOP version 1.2 (C) 2009 Intel Corporation Collecting data for 5.00 second(s) root@lab10:~# vi /tmp/powertop.5.1 root@lab10:~# cat /tmp/powertop.5.1 OpenIndiana PowerTOP version 1.2 C-states (idle power) Avg Residency P-states (frequencies) C0 (cpu running) (9.9%) 1600 Mhz 0.0% C1 0.0ms (0.0%) 1800 Mhz 0.0% C2 0.0ms (0.0%) 2000 Mhz 0.0% C3 2.4ms (90.1%) 2200 Mhz 0.0% 2400 Mhz 0.0% 2600 Mhz 0.0% 2800 Mhz 0.0% 3000 Mhz 0.0% 3200 Mhz 0.0% 3400 Mhz 0.0% 3800 Mhz(turbo) 100.0% Wakeups-from-idle per second: 3103.9 interval: 5.0s no ACPI power usage estimate available Top causes for wakeups: 91.8% (2848.9) sched : unix`dtrace_xcall_func 3.2% (100.2) : genunix`clock 2.5% ( 76.2) : genunix`cv_wakeup 1.6% ( 50.0) : SDC`sysdc_update 0.3% ( 8.0) : ehci`ehci_handle_root_hub_status_change 0.3% ( 8.0) : cpudrv`cpudrv_monitor_disp 0.1% ( 4.0) : genunix`schedpaging 0.1% ( 2.2) : e1000g#0 0.0% ( 1.0) : TS`ts_update 0.0% ( 1.0) : e1000g`e1000g_local_timer 0.0% ( 1.0) : genunix`sigalarm2proc 0.0% ( 1.0) : ehci#0 0.0% ( 1.0) : ehci#1 0.0% ( 0.2) : genunix`vmem_update 0.0% ( 0.2) : ip`tcp_timer_callback 0.0% ( 0.2) : swrand`rnd_handler 0.0% ( 0.2) sched : unix`hati_demap_func 0.0% ( 0.2) : genunix`kmem_update 0.0% ( 0.2) : kcf`rnd_handler 0.0% ( 0.2) : ahci`ahci_watchdog_handler Suggestion: enable CPU power management by pressing the P key root@lab10:~# root@lab10:~# prtdiag System Configuration: Supermicro X9SCI/X9SCA BIOS Configuration: American Megatrends Inc. 4.6.4 02/24/2011 BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style) ==== Processor Sockets ==================================== Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU E31270 @ 3.40GHz SOCKET 0 ==== Memory Device Sockets ================================ Type Status Set Device Locator Bank Locator ----------- ------ --- ------------------- ---------------- Unknown in use 0 DIMM_1A BANK0 Unknown in use 0 DIMM_2A BANK0 Unknown in use 0 DIMM_1B BANK0 Unknown in use 0 DIMM_2B BANK0 ==== On-Board Devices ===================================== To Be Filled By O.E.M. ==== Upgradeable Slots ==================================== ID Status Type Description --- --------- ---------------- ---------------------------- 0 in use PCI Express J6B2 1 in use PCI Express J6B1 2 in use PCI Express J6D1 3 in use PCI Express J7B1 4 in use PCI Express J8B4 root@lab10:~# cat /etc/power.conf # # Copyright 1996-2002 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # #pragma ident "@(#)power.conf 2.1 02/03/04 SMI" # # Power Management Configuration File # cpu_deep_idle enable cpu-threshold 10s cpupm disable cpupm disable root@lab10:~# dtrace -n 'profile-10ms{@[stack()] = count()}' dtrace: description 'profile-10ms' matched 1 probe ^C unix`mutex_enter+0x10 unix`hment_mapcnt+0x1d unix`hat_page_getshare+0x16 genunix`fsflush_do_pages+0x236 genunix`fsflush+0x39a unix`thread_start+0x8 1 unix`x86pte_mapin+0x47 unix`x86pte_access_pagetable+0x3d unix`x86pte_set+0x8d unix`hati_pte_map+0x187 unix`hati_load_common+0x15d unix`hat_memload+0x81 unix`segkmem_xalloc+0x13b unix`segkmem_alloc_vn+0xcd unix`segkmem_alloc+0x24 genunix`vmem_xalloc+0x546 genunix`vmem_alloc+0x161 genunix`kmem_alloc+0x64 genunix`kmem_zalloc+0x3b autofs`auto_calldaemon+0x22d autofs`auto_send_unmount_request+0x6b autofs`unmount_node+0x181 autofs`try_unmount_node+0x13b autofs`unmount_subtree+0x79 autofs`unmount_tree+0xcf autofs`unmount_zone_tree+0x1e 1 genunix`fsflush_do_pages+0x1da genunix`fsflush+0x39a unix`thread_start+0x8 1 unix`page_pptonum+0xc genunix`fsflush_do_pages+0x35d genunix`fsflush+0x39a unix`thread_start+0x8 1 7 unix`i86_mwait+0xd unix`cpu_idle_mwait+0xf1 unix`cpu_acpi_idle+0x8d unix`cpu_idle_adaptive+0x19 unix`idle+0x114 unix`thread_start+0x8 10 unix`acpi_cpu_cstate+0x2ae unix`cpu_acpi_idle+0x82 unix`cpu_idle_adaptive+0x19 unix`idle+0x114 unix`thread_start+0x8 10027 root@lab10:~# root@lab10:~# echo ::interrupts | mdb -k CPU/Vect IRQ IPL Bus Trg Type Share APIC/INT# ISR 0/0x25 16 9 PCI Lvl Fixed 1 0x0/0x10 ehci_intr 1/0x20 9 9 PCI Lvl Fixed 1 0x0/0x9 acpi_wrapper_isr 1/0x21 23 9 PCI Lvl Fixed 1 0x0/0x17 ehci_intr 1/0x22 4 12 ISA Edg Fixed 1 0x0/0x4 asyintr 2/0x20 11 14 PCI Lvl Fixed 1 0x0/0xb hpet_isr 2/0x21 - 6 PCI Edg MSI 1 - e1000g_intr_pciexpress 3/0x20 - 5 PCI Edg MSI 1 - ahci_intr 3/0x21 3 12 ISA Edg Fixed 1 0x0/0x3 asyintr 5/0x20 10 12 ISA Edg Fixed 1 0x0/0xa asyintr 6/0x20 1 5 ISA Edg Fixed 1 0x0/0x1 i8042_intr 6/0x21 - 6 PCI Edg MSI 1 - e1000g_intr_pciexpress 7/0x20 12 5 ISA Edg Fixed 1 0x0/0xc i8042_intr all/0xf0 - 15 - Edg IPI 1 - xc_serv all/0xf1 - 11 - Edg IPI 0 - poke_cpu all/0xf2 - 14 - Edg IPI 1 - kcpc_hw_overflow_intr all/0xf3 - 15 - Edg IPI 1 - apic_error_intr all/0xf4 - 2 - Edg IPI 1 - cmi_cmci_trap all/0xf5 - 14 - Edg IPI 1 - cbe_fire all/0xf6 - 14 - Edg IPI 1 - cbe_fire root@lab10:~# echo ::interrupts -d | mdb -k CPU/Vect IRQ IPL Bus Trg Type Share APIC/INT# Driver Name(s) 0/0x25 16 9 PCI Lvl Fixed 1 0x0/0x10 ehci#0 1/0x20 9 9 PCI Lvl Fixed 1 0x0/0x9 acpi_wrapper_isr 1/0x21 23 9 PCI Lvl Fixed 1 0x0/0x17 ehci#1 1/0x22 4 12 ISA Edg Fixed 1 0x0/0x4 asy#0 2/0x20 11 14 PCI Lvl Fixed 1 0x0/0xb hpet_isr 2/0x21 - 6 PCI Edg MSI 1 - e1000g#0 3/0x20 - 5 PCI Edg MSI 1 - ahci#0 3/0x21 3 12 ISA Edg Fixed 1 0x0/0x3 asy#1 5/0x20 10 12 ISA Edg Fixed 1 0x0/0xa asy#2 6/0x20 1 5 ISA Edg Fixed 1 0x0/0x1 i8042#0 6/0x21 - 6 PCI Edg MSI 1 - e1000g#1 7/0x20 12 5 ISA Edg Fixed 1 0x0/0xc i8042#0 all/0xf0 - 15 - Edg IPI 1 - xc_serv all/0xf1 - 11 - Edg IPI 0 - poke_cpu all/0xf2 - 14 - Edg IPI 1 - kcpc_hw_overflow_intr all/0xf3 - 15 - Edg IPI 1 - apic_error_intr all/0xf4 - 2 - Edg IPI 1 - cmi_cmci_trap all/0xf5 - 14 - Edg IPI 1 - cbe_fire all/0xf6 - 14 - Edg IPI 1 - cbe_fire root@lab10:~# root@lab10:~# intrstat 30 device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim -------------+------------------------------------------------------------------------------------------------------------------------ e1000g#0 | 0 0.0 0 0.0 1 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 ^C root@lab10:~# root@lab10:~# powertop -t 30 OpenIndiana PowerTOP version 1.2 (C) 2009 Intel Corporation Collecting data for 30.00 second(s) root@lab10:~# vi /tmp/powertop.30.1 root@lab10:~# cat /tmp/powertop.30.1 OpenIndiana PowerTOP version 1.2 C-states (idle power) Avg Residency P-states (frequencies) C0 (cpu running) (10.4%) 1600 Mhz 0.0% C1 0.0ms (0.0%) 1800 Mhz 0.0% C2 0.0ms (0.0%) 2000 Mhz 0.0% C3 2.4ms (89.6%) 2200 Mhz 0.0% 2400 Mhz 0.0% 2600 Mhz 0.0% 2800 Mhz 0.0% 3000 Mhz 0.0% 3200 Mhz 0.0% 3400 Mhz 0.0% 3711 Mhz(turbo) 100.0% Wakeups-from-idle per second: 3102.0 interval: 30.0s no ACPI power usage estimate available Top causes for wakeups: 91.8% (2848.9) sched : unix`dtrace_xcall_func 3.2% (100.0) : genunix`clock 2.5% ( 77.2) : genunix`cv_wakeup 1.6% ( 50.0) : SDC`sysdc_update 0.3% ( 8.0) : cpudrv`cpudrv_monitor_disp 0.2% ( 7.7) : ehci`ehci_handle_root_hub_status_change 0.1% ( 4.0) : genunix`schedpaging 0.0% ( 1.3) : e1000g#0 0.0% ( 1.0) : TS`ts_update 0.0% ( 1.0) : e1000g`e1000g_local_timer 0.0% ( 1.0) : ehci#1 0.0% ( 1.0) : ehci#0 0.0% ( 0.2) : kcf`rnd_handler 0.0% ( 0.2) : ahci`ahci_watchdog_handler 0.0% ( 0.2) : swrand`rnd_handler 0.0% ( 0.1) sched : unix`hati_demap_func 0.0% ( 0.1) : ip`igmp_slowtimo 0.0% ( 0.1) : genunix`vmem_update 0.0% ( 0.1) : genunix`kmem_update 0.0% ( 0.0) : ip`tcp_timer_callback Suggestion: enable CPU power management by pressing the P key root@lab10:~# root@lab10:~# top -s 10 last pid: 3376; load avg: 0.01, 0.01, 0.00; up 6+15:48:02 07:28:25 62 processes: 61 sleeping, 1 on cpu CPU states: 93.4% idle, 0.1% user, 6.5% kernel, 0.0% iowait, 0.0% swap Kernel: 319 ctxsw, 3 trap, 963 intr, 61 syscall, 2 flt Memory: 16G phys mem, 14G free mem, 8187M total swap, 8187M free swap PID USERNAME NLWP PRI NICE SIZE RES STATE TIME CPU COMMAND 1204 root 3 58 0 59M 36M sleep 31:49 0.04% Xorg 1657 gdm 1 59 0 87M 25M sleep 3:12 0.00% gdm-simple-gree 3375 root 1 59 0 3852K 2492K cpu/3 0:00 0.00% top 1713 root 21 59 0 8428K 5332K sleep 0:04 0.00% nscd 1733 admin 1 59 0 7776K 5188K sleep 0:01 0.00% sshd 1642 gdm 1 59 0 82M 18M sleep 0:04 0.00% gnome-power-man 264 root 6 59 0 4356K 3156K sleep 0:02 0.00% devfsadm 1057 root 6 59 0 26M 10M sleep 0:02 0.00% fbserver 761 root 26 59 0 18M 12M sleep 0:02 0.00% fmd 10 root 16 59 0 16M 13M sleep 0:02 0.00% svc.startd 12 root 22 59 0 16M 14M sleep 0:07 0.00% svc.configd 755 root 4 59 0 6728K 3872K sleep 0:00 0.00% inetd 45 netcfg 5 59 0 4660K 3660K sleep 0:01 0.00% netcfgd 751 root 1 59 0 1620K 916K sleep 0:00 0.00% utmpd 894 root 1 59 0 6968K 5852K sleep 0:15 0.00% intrd 1726 root 4 59 0 3068K 1948K sleep 0:08 0.00% automountd 1732 root 1 59 0 7152K 4536K sleep 0:04 0.00% sshd 372 root 4 59 0 6912K 5548K sleep 0:03 0.00% hald 401 root 1 59 0 3496K 2092K sleep 0:03 0.00% hald-addon-acpi 1611 gdm 1 59 0 75M 13M sleep 0:01 0.00% metacity 1596 gdm 1 59 0 7828K 6020K sleep 0:01 0.00% at-spi-registry 1594 gdm 1 59 0 7152K 5948K sleep 0:01 0.00% gconfd-2 753 root 11 59 0 4104K 2096K sleep 0:01 0.00% syslogd 238 root 5 60 -20 2548K 1512K sleep 0:00 0.00% zonestatd 1597 gdm 1 59 0 99M 35M sleep 0:00 0.00% gnome-settings- 1581 gdm 2 59 0 15M 10M sleep 0:00 0.00% gnome-session 885 root 4 59 0 10M 6652K sleep 0:00 0.00% rad 289 root 1 59 0 11M 5684K sleep 0:00 0.00% cupsd 1599 gdm 2 59 0 7140K 5120K sleep 0:00 0.00% bonobo-activati 94 root 8 59 0 6320K 5108K sleep 0:00 0.00% nwamd