Bug #3084
System slow when accssing many file using CIFS.
Status:
Feedback
Priority:
Low
Assignee:
-
Category:
cifs - CIFS server and client
Start date:
2012-08-15
Due date:
% Done:
0%
Estimated time:
Difficulty:
Hard
Tags:
needs-triage
Gerrit CR:
Description
General Description
The entire system becomes 'slow' during(possible also after) accessing many files in a large CIFS share.
I am sometimes able to trigger this when for example using 7zip to unpack a large multi part rar archive.
Or when windows media center starts to index a large share for all video/music content.¶
Symptoms
I would generally describe it as system is slow. Although this is not 100% accurate.
Here is an example of running ls in this state, this usually happens on the first attempt, consecutive attempts sometimes work normal or show the same symptom in a lesser degree.¶
@gravity ~ # time ls
scripts TODO
real 0m18.079s
user 0m0.001s
sys 0m0.003s@
More information
tsoome helped me nail it down, below is a bunch of pastes of stuff I looked up for myself or for him.
http://pastebin.com/RUbBP4f4
http://paste.ubuntu.com/1148488/
There were more but I recently clean installed OSX Mountain Lion so I lost them.¶
Some more 'experience' description
Through my poking around and looking at things, I've gotten the feeling that the system is not 'slow' it self. Once for example heavy task gets started... it works as expected without any slowdowns.¶
But there is a huge delay before this task get started.
A nice example is a ssh login in this state can sometimes take 10-20 minutes!
I recently did notice that removing /usr/bin/quota from /etc/profiles, reduces this significantly.
lost test data so this should probably be redone at some point:
Initial assumption was that it was my network setup, I swapped around the nics in the aggr's or tried without aggr. Same effect.¶
Disk latency, local disk intensive tasks have no issue... even in the 'slow' state the work fine once the get started.
Interupts, nothing out of the ordinary
Sysload/idle, again all seemed fine on this front.
Heat issues, happens on both hot days and cold days.
Running for example a stress test tool on a warm then, then booting the system normally, did not trigger the effect.
I also stripped the system of all unneeded components (PCI nics and one of the HBA's, cut down the ram to 16gb. swapping various modules along the way)
Updates all the firmwares and drivers to latest version.
system hardware and configuration
I'll be brief here, most of this is covered in my personal wiki.
Although this is under the personal section:
https://docu.blackdot.be/personal/gravity/planning (hardware)
https://docu.blackdot.be/personal/gravity/notes (configuration)
Login: illumos with password illumos¶
Sorry for not moving it to a public section, I stripped most of my super sensitive information. But I would still like to keep it away from the general public.
The information there is fairly accurate. I did recently switch to zsh but the problem is happening from before this.
Although I run OpenIndiana, OmniOS seems to have the same issue.
I did a clean install (without mirroring the OS), create the user accounts for CIFS and imported my zpools.
I managed to hit the problem with that system too after playing around with windows media center's indexing.
History
Updated by Jorge Schrauwen almost 8 years ago
Updated by - Status changed from New to Feedback
Here is some fresh data from this morning:
-(~)-[✓]-{ time truss ls }-(sjorge@gravity)-
execve("/usr/gnu/bin/ls", 0x08047D7C, 0x08047D84) argc = 1
sysinfo(SI_MACHINE, "i86pc", 257) = 6
mmap(0x00000000, 32, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFB0000
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFA0000
sysconfig(_CONFIG_PAGESIZE) = 4096
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF90000
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF80000
memcntl(0xFEFB7000, 32112, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
memcntl(0x08050000, 21548, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12
resolvepath("/usr/gnu/bin/ls", "/usr/gnu/bin/ls", 1023) = 15
stat64("/usr/gnu/bin/ls", 0x080479C0) = 0
open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT
stat64("/lib/libsec.so.1", 0x08047170) = 0
resolvepath("/lib/libsec.so.1", "/lib/libsec.so.1", 1023) = 16
open("/lib/libsec.so.1", O_RDONLY) = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFEFA0D78, 0x080471DC, 0x00000000) = 0
close(3) = 0
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF50000
memcntl(0xFEF60000, 15192, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
stat64("/lib/libc.so.1", 0x08047170) = 0
resolvepath("/lib/libc.so.1", "/lib/libc.so.1", 1023) = 14
open("/lib/libc.so.1", O_RDONLY) = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFEF50550, 0x080471DC, 0x00000000) = 0
close(3) = 0
memcntl(0xFEE10000, 179176, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEE00000
mmap(0x00010000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEDF0000
getcontext(0x08047820)
getrlimit(RLIMIT_STACK, 0x08047818) = 0
getpid() = 3756 [3755]
lwp_private(0, 1, 0xFEDF2A40) = 0x000001C3
setustack(0xFEDF2AA0)
sysi86(SI86FPSTART, 0xFEF49C24, 0x0000133F, 0x00001F80) = 0x00000001
open("/usr/lib/locale//en_US.UTF-8/LC_CTYPE/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475B0) = 0
fstat(3, 0x08047710) = 0
brk(0x08083910) = 0
brk(0x0808B910) = 0
llseek(3, 0, SEEK_CUR) = 0
llseek(3, 0, SEEK_SET) = 0
fstat64(3, 0x080475A0) = 0
brk(0x0808B910) = 0
brk(0x08093910) = 0
fstat64(3, 0x080474B0) = 0
ioctl(3, TCGETA, 0x08047550) Err#25 ENOTTY
read(3, " R u n e M a g 1 U T F -".., 30208) = 30188
brk(0x08093910) = 0
brk(0x0809D910) = 0
llseek(3, 0, SEEK_CUR) = 30188
close(3) = 0
open("/usr/lib/locale//en_US.UTF-8/LC_NUMERIC/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x08047590) = 0
fstat(3, 0x08047B60) = 0
read(3, " .\n ,\n 3\n", 6) = 6
close(3) = 0
open("/usr/lib/locale//en_US.UTF-8/LC_TIME/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475A0) = 0
fstat(3, 0x08047B70) = 0
read(3, " J a n\n F e b\n M a r\n".., 308) = 308
close(3) = 0
open("/usr/lib/locale//en_US.UTF-8/LC_COLLATE/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475C0) = 0
fstat(3, 0x08047BA0) = 0
mmap(0x00000000, 75408, PROT_READ, MAP_PRIVATE, 3, 0) = 0xFEDDC000
close(3) = 0
open("/usr/lib/locale//en_US.UTF-8/LC_MONETARY/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x08047590) = 0
fstat(3, 0x08047B60) = 0
read(3, " U S D \n $\n .\n ,\n 3".., 44) = 44
close(3) = 0
open("/usr/lib/locale//en_US.UTF-8/LC_MESSAGES/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475A0) = 0
fstat(3, 0x08047B70) = 0
read(3, " ^ ( ( [ y Y ] ( [ e E ]".., 45) = 45
close(3) = 0
ioctl(1, TCGETA, 0x08047C50) = 0
ioctl(1, TIOCGWINSZ, 0x08047CB8) = 0
open(".", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3
fstat64(3, 0x08047940) = 0
fcntl(3, F_SETFD, 0x00000001) = 0
fstat64(3, 0x08047A40) = 0
getdents64(3, 0xFEDF4000, 8192) = 528
getdents64(3, 0xFEDF4000, 8192) = 0
close(3) = 0
close(1) = 0
close(2) = 0
_exit(0)
truss ls 0.00s user 0.00s system 0% cpu 26.226 total
-(~)-[✓]-{ vmstat 5 5 }-(sjorge@gravity)-
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s3 s9 s1 in sy cs us sy id
1 0 0 34207128 21780212 7 34 0 0 0 0 26 3 -0 1 1 1436 868 1100 0 0 100
0 0 0 30331980 17890896 7 14 0 0 0 0 0 0 0 4 4 1269 375 949 0 0 100
1 0 0 30331868 17890816 0 0 0 0 0 0 0 0 0 0 0 1087 368 814 0 0 100
1 0 0 30331868 17890816 0 0 0 0 0 0 0 0 0 0 0 1092 405 824 0 0 100
0 0 0 30331868 17890816 0 0 0 0 0 0 0 0 0 0 0 1141 354 808 0 0 100
-(~)-[✓]-{ mpstat 5 5 }-(sjorge@gravity)-
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 2 0 3 455 219 85 0 2 4 0 61 0 0 0 99
1 5 0 8 113 33 134 0 2 4 0 103 0 0 0 99
2 5 0 5 96 25 110 0 2 3 0 95 0 0 0 99
3 2 0 3 79 23 85 0 1 3 0 47 0 0 0 100
4 2 0 2 61 14 64 0 1 3 0 67 0 0 0 100
5 3 0 5 86 21 100 0 2 3 0 83 0 0 0 100
6 4 0 5 96 25 109 0 2 3 0 82 0 0 0 100
7 1 0 2 101 53 67 0 1 5 0 49 0 0 0 100
8 1 0 2 60 15 61 0 1 2 0 39 0 0 0 100
9 4 0 5 100 31 109 0 2 4 0 109 0 0 0 100
10 5 0 6 104 35 109 0 2 4 0 90 0 0 0 100
11 1 0 2 83 36 65 0 1 3 0 37 0 0 0 100
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 333 110 56 0 2 1 0 17 0 0 0 100
1 0 0 7 85 20 96 0 3 4 0 40 0 0 0 100
2 0 0 0 85 30 90 0 3 6 0 46 0 1 0 99
3 0 0 9 53 29 34 0 2 3 0 17 0 0 0 100
4 0 0 0 86 32 94 0 1 2 0 30 0 0 0 100
5 0 0 9 127 55 136 0 2 5 0 16 0 0 0 100
6 1 0 0 67 5 59 0 3 2 0 71 0 0 0 100
7 0 0 0 89 27 77 0 1 2 0 89 0 0 0 100
8 0 0 2 96 27 83 0 1 2 0 27 0 0 0 100
9 0 0 9 60 26 89 0 1 4 0 90 0 0 0 100
10 0 0 2 65 16 60 0 1 4 0 41 0 0 0 100
11 0 0 0 51 11 38 0 1 1 0 10 0 0 0 100
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 315 105 24 0 0 0 0 6 0 0 0 100
1 0 0 0 49 9 41 0 1 0 0 23 0 0 0 100
2 0 0 0 53 12 47 0 1 0 0 16 0 1 0 99
3 0 0 0 42 9 40 0 1 0 0 8 0 0 0 100
4 0 0 0 63 19 60 0 2 0 0 5 0 0 0 100
5 0 0 4 125 54 121 0 1 0 0 9 0 0 0 100
6 1 0 4 58 12 59 0 1 0 0 89 0 0 0 100
7 0 0 0 35 3 25 0 1 0 0 11 0 0 0 100
8 0 0 0 130 48 119 0 0 0 0 47 0 0 0 100
9 0 0 4 96 42 103 0 1 0 0 60 0 0 0 100
10 0 0 4 129 49 135 0 1 1 0 83 0 0 0 100
11 0 0 0 32 6 19 0 0 0 0 3 0 0 0 100
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 4 331 105 72 0 2 0 0 20 0 0 0 100
1 141 0 9 74 19 77 0 3 1 0 110 0 0 0 100
2 0 0 12 94 31 96 0 2 0 0 38 0 0 0 99
3 0 0 0 66 16 58 0 2 1 0 22 0 0 0 100
4 0 0 11 51 7 43 0 1 1 0 11 0 1 0 99
5 69 0 11 93 35 81 0 1 1 0 35 0 0 0 100
6 80 0 4 48 4 32 0 1 1 0 103 0 0 0 100
7 0 0 0 28 2 14 0 0 0 0 4 0 0 0 100
8 0 0 18 124 50 111 0 0 0 0 49 0 0 0 100
9 0 0 9 91 35 78 0 1 0 0 27 0 0 0 100
10 232 0 18 126 42 128 0 1 1 0 798 1 0 0 99
11 0 0 0 39 5 20 0 1 0 0 4 0 0 0 100
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 0 0 0 331 105 45 0 1 2 0 12 0 0 0 100
1 0 0 7 92 26 110 0 1 3 0 66 0 0 0 100
2 0 0 4 38 7 38 0 2 3 0 9 0 1 0 99
3 0 0 13 76 34 58 0 1 1 0 34 0 0 0 100
4 0 0 0 47 3 37 0 1 1 0 5 0 0 0 100
5 0 0 4 63 17 62 0 2 2 0 15 0 0 0 100
6 0 0 0 72 16 69 0 3 2 0 89 0 0 0 100
7 0 0 11 46 10 42 0 2 3 0 16 0 0 0 100
8 0 0 13 152 59 141 0 2 1 0 52 0 0 0 100
9 0 0 18 208 95 200 0 1 1 0 47 0 0 0 100
10 0 0 4 81 15 65 0 1 3 0 10 0 0 0 100
11 0 0 0 28 3 15 0 1 1 0 4 0 0 0 100
-(~)-[✗]-{ intrstat }-(root@gravity)-
device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.2
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#3 | 1 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.2
device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim
-------------+------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 1 0.0
e1000g#1 | 0 0.0 15 0.0 0 0.0 0 0.0
e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 12 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0
device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 2 0.0 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#3 | 2 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0
device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim
-------------+------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 2 0.0
e1000g#1 | 0 0.0 22 0.0 0 0.0 0 0.0
e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 16 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
ehci#1 | 1 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0
device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#3 | 1 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim
-------------+------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 1 0.0
e1000g#1 | 0 0.0 18 0.0 0 0.0 0 0.0
e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 16 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0
device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 88 0.1 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#3 | 1 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0
device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim
-------------+------------------------------------------------------------
ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0
arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#0 | 0 0.0 0 0.0 0 0.0 1 0.0
e1000g#1 | 0 0.0 21 0.0 0 0.0 0 0.0
e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0
e1000g#4 | 0 0.0 0 0.0 15 0.0 0 0.0
ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0
ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0
nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0
-(DTT)-[✗]-{ dtrace -n 'syscall:::entry { @num[probefunc] = count(); }' }-(root@gravity)-
dtrace: description 'syscall:::entry ' matched 231 probes
^X^C
getloadavg 1
kill 1
lwp_detach 1
lwp_mutex_timedlock 1
lwp_mutex_unlock 1
mmap 1
setsockopt 1
shutdown 1
sigpending 1
sendto 2
umount2 2
statvfs64 3
sysconfig 3
alarm 4
fstat 4
getsockname 4
pause 4
recvfrom 4
sigaction 4
openat 6
putmsg 6
rename 6
brk 8
accept 11
getdents64 12
lwp_continue 13
lwp_create 13
lwp_exit 13
uucopy 13
pread 14
schedctl 14
clock_gettime 16
lwp_kill 17
getpeername 18
unlink 18
send 23
c2audit 25
portfs 25
getuid 36
writev 37
getsockopt 40
connect 42
so_socket 49
open 62
stat 71
fdsync 72
open64 72
getmsg 75
stat64 76
sendmsg 92
recvmsg 95
pset 141
pwrite64 180
close 225
waitsys 237
sigsuspend 238
p_online 256
setcontext 264
nanosleep 276
pread64 503
access 960
lwp_sigmask 960
read 1103
write 1177
llseek 1210
fstat64 1546
doorfs 1614
getpid 1667
fcntl 2304
gtime 2422
recv 5302
lwp_park 23651
pollsys 44133
ioctl 64358
(DTT)-[✓]-{ ./hotkernel }-(root@gravity)-
Sampling... Hit Ctrl-C to end.
^C
FUNCTION COUNT PCNT
unix`page_unlock 1 0.0%
genunix`thread_free_prevent 1 0.0%
aggr`aggr_find_tx_ring 1 0.0%
unix`gdt_ucode_model 1 0.0%
arcsas`arcsas_build_os_ccb 1 0.0%
genunix`new_mstate 1 0.0%
genunix`clock_tick 1 0.0%
SDC`sysdc_wakeup 1 0.0%
genunix`callout_list_get 1 0.0%
unix`xsave_ctxt 1 0.0%
unix`reset_turbo_info 1 0.0%
zfs`zio_vdev_io_done 1 0.0%
genunix`taskq_ent_alloc 1 0.0%
sha1`sha1_block_data_order 1 0.0%
sd`sdstrategy 1 0.0%
genunix`kmem_cache_alloc 1 0.0%
unix`set_freemem 1 0.0%
unix`cpupm_wakeup_cstate_data 1 0.0%
genunix`pcache_poll 1 0.0%
genunix`callout_list_expire 1 0.0%
genunix`lwp_hash_lookup_and_lock 1 0.0%
unix`rw_exit 1 0.0%
stmf`stmf_svc_timeout 1 0.0%
arcsas`arcsas_device_get_sent_ccb_from_list 1 0.0%
zfs`vdev_queue_io_remove 1 0.0%
rootnex`rootnex_coredma_allochdl 1 0.0%
genunix`copyin_args32 1 0.0%
genunix`callout_downheap 1 0.0%
zfs`vdev_queue_offset_compare 1 0.0%
genunix`setrun_locked 1 0.0%
unix`strcpy 1 0.0%
unix`cpu_resched 1 0.0%
unix`_sys_sysenter_post_swapgs 1 0.0%
unix`bcopy 1 0.0%
genunix`disp_lock_exit_high 1 0.0%
genunix`avl_walk 1 0.0%
unix`av_dispatch_softvect 1 0.0%
zfs`dmu_object_size_from_db 1 0.0%
TS`ts_sleep 1 0.0%
genunix`mexchange 1 0.0%
arcsas`arcsas_ccb_done 1 0.0%
genunix`callout_expire 1 0.0%
ip`squeue_worker 1 0.0%
arcsas`arcsas_check_irq_state 1 0.0%
zfs`buf_hash_find 1 0.0%
unix`sep_restore 1 0.0%
unix`prunstop 1 0.0%
sd`sdintr 1 0.0%
unix`xc_call 1 0.0%
zfs`arc_buf_remove_ref 1 0.0%
genunix`syscall_exit 1 0.0%
zfs`space_map_seg_compare 1 0.0%
genunix`cyclic_reprogram_cyclic 1 0.0%
genunix`taskq_thread 1 0.0%
pcplusmp`apic_unset_idlecpu 1 0.0%
genunix`cv_waituntil_sig 1 0.0%
genunix`schedctl_set_cidpri 1 0.0%
unix`x86_hm_exit 1 0.0%
genunix`cv_wait 1 0.0%
genunix`fd_find 1 0.0%
unix`cpupm_utilization_event 1 0.0%
zfs`dbuf_hash 1 0.0%
pcplusmp`apic_send_ipi 1 0.0%
unix`cpupm_state_change 1 0.0%
genunix`schedctl_restore 1 0.0%
genunix`ddi_dma_sync 1 0.0%
unix`atomic_add_64_nv 1 0.0%
genunix`list_remove 1 0.0%
zfs`dnode_block_freed 1 0.0%
genunix`avl_numnodes 1 0.0%
unix`hat_kpm_pfn2va 1 0.0%
genunix`sleepq_insert 1 0.0%
genunix`expire_tick 1 0.0%
unix`idle_exit 1 0.0%
unix`av_clear_softint_pending 1 0.0%
genunix`cv_wait_sig_swap_core 1 0.0%
unix`rdmsr 1 0.0%
genunix`turnstile_block 1 0.0%
genunix`cv_unsleep 1 0.0%
genunix`sys_nullsys 1 0.0%
unix`clock_tick_process 1 0.0%
arcsas`arcsas_set_lba_and_sector_count 1 0.0%
unix`atomic_add_64 1 0.0%
unix`atomic_add_32_nv 1 0.0%
unix`rw_enter 2 0.0%
genunix`poll_common 2 0.0%
unix`gdt_update_usegd 2 0.0%
genunix`cpu_update_pct 2 0.0%
unix`hat_switch 2 0.0%
unix`lock_try 2 0.0%
unix`wrmsr 2 0.0%
genunix`cyclic_softint 2 0.0%
unix`sep_save 2 0.0%
unix`atomic_cas_32 2 0.0%
unix`setbackdq 2 0.0%
genunix`untimeout_default 2 0.0%
genunix`thread_lock 2 0.0%
unix`cpu_acpi_idle 2 0.0%
unix`bitset_atomic_add 2 0.0%
unix`cpupm_next_cstate 2 0.0%
unix`_resume_from_idle 2 0.0%
genunix`ddi_get_lbolt 2 0.0%
unix`clock_tick_execute_common 2 0.0%
unix`lwp_segregs_restore32 2 0.0%
genunix`cv_timedwait_sig_hires 2 0.0%
unix`lock_set_spl 2 0.0%
genunix`taskq_dispatch 2 0.0%
mac`mac_rx_deliver 2 0.0%
genunix`callout_heap_delete 3 0.0%
unix`mutex_delay_default 3 0.0%
unix`mul32 3 0.0%
zfs`l2arc_write_buffers 3 0.0%
unix`0xfffffffffb85 3 0.0%
unix`disp_anywork 3 0.0%
unix`mutex_tryenter 3 0.0%
genunix`clock 3 0.0%
unix`default_lock_delay 3 0.0%
unix`tsc_scalehrtime 3 0.0%
unix`cmt_ev_thread_swtch 3 0.0%
unix`atomic_and_64 3 0.0%
unix`splr 3 0.0%
unix`disp_ratify 3 0.0%
genunix`syscall_entry 3 0.0%
genunix`timeout_generic 3 0.0%
dtrace`dtrace_dynvar_clean 3 0.0%
rootnex`rootnex_teardown_copybuf 3 0.0%
unix`bzero 3 0.0%
genunix`post_syscall 4 0.0%
unix`cmt_balance 4 0.0%
unix`mutex_owned 4 0.0%
unix`mutex_exit 4 0.0%
unix`page_pptonum 4 0.0%
unix`lock_set 4 0.0%
unix`idle 4 0.0%
unix`bitset_in_set 4 0.0%
sha2`SHA256TransformBlocks 5 0.0%
unix`swtch 5 0.0%
unix`resume 5 0.0%
unix`dispatch_hilevel 5 0.0%
unix`ddi_get32 5 0.0%
unix`membar_enter 5 0.0%
unix`prefetch_page_r 6 0.0%
genunix`syscall_mstate 6 0.0%
genunix`restorectx 6 0.0%
unix`av_check_softint_pending 7 0.0%
unix`atomic_add_32 8 0.0%
zfs`lzjb_compress 9 0.0%
unix`atomic_or_64 9 0.0%
unix`tsc_gethrtime_delta 10 0.0%
zfs`l2arc_write_eligible 10 0.0%
genunix`lbolt_event_driven 11 0.0%
unix`mutex_enter 13 0.0%
genunix`list_next 14 0.0%
unix`disp_getwork 18 0.0%
unix`tsc_gethrtimeunscaled_delta 19 0.0%
unix`dispatch_softint 26 0.0%
zfs`buf_hash 27 0.0%
unix`do_splx 47 0.0%
unix`page_nextn 103 0.0%
genunix`fsflush_do_pages 108 0.0%
unix`i86_mwait 601 0.1%
unix`acpi_cpu_cstate 405605 99.7%
-(~adm)-[✓]-{ tail -n200 messages }-(root@gravity)-
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hubd1 is /pci@0,0/pci8086,4953@1d/hub@1
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1d/hub@1 (hubd1) online
Aug 15 20:09:47 gravity.acheron.be drmach_acpi: [ID 779328 kern.warning] WARNING: drmach_init: number of logical CPUs (12) in physical processor is not power of 2.
Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 device (usb4f3,103) operating at low speed (USB 1.x) on USB 2.0 external hub: device@5, usb_mid1 at bus address 3
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] usb_mid1 is /pci@0,0/pci8086,4953@1a/hub@1/device@5
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5 (usb_mid1) online
Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 interface (usbif4f3,103.config1.0) operating at low speed (USB 1.x) on USB 2.0 external hub: keyboard@0, hid4 at bus address 3
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hid4 is /pci@0,0/pci8086,4953@1a/hub@1/device@5/keyboard@0
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5/keyboard@0 (hid4) online
Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 interface (usbif4f3,103.config1.1) operating at low speed (USB 1.x) on USB 2.0 external hub: input@1, hid5 at bus address 3
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hid5 is /pci@0,0/pci8086,4953@1a/hub@1/device@5/input@1
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5/input@1 (hid5) online
Aug 15 20:09:48 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: audio0
Aug 15 20:09:48 gravity.acheron.be genunix: [ID 936769 kern.info] audio0 is /pseudo/audio@0
Aug 15 20:09:51 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10d3 (e1000g) instance 1 irq 0x34 vector 0x60 ioapic 0xff intin 0xff is bound to cpu 9
Aug 15 20:09:51 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g1 registered
Aug 15 20:09:51 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,107d (e1000g) instance 4 irq 0x35 vector 0x61 ioapic 0xff intin 0xff is bound to cpu 10
Aug 15 20:09:51 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g4 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: aggr1000 registered
Aug 15 20:09:52 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pci8086,1502 (e1000g) instance 0 irq 0x36 vector 0x62 ioapic 0xff intin 0xff is bound to cpu 11
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g0 registered
Aug 15 20:09:52 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10d3 (e1000g) instance 3 irq 0x37 vector 0x63 ioapic 0xff intin 0xff is bound to cpu 0
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g3 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: aggr1001 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1002 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1003 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1002 link up, 0 Mbps, unknown duplex
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1004 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1005 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1006 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1007 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1008 registered
Aug 15 20:09:53 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Aug 15 20:09:53 gravity.acheron.be genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Aug 15 20:09:54 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: zfs0
Aug 15 20:09:54 gravity.acheron.be genunix: [ID 936769 kern.info] zfs0 is /pseudo/zfs@0
Aug 15 20:09:54 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g4 link up, 1000 Mbps, full duplex
Aug 15 20:09:54 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: aggr1000 link up, 1000 Mbps, full duplex
Aug 15 20:09:54 gravity.acheron.be drmach_acpi: [ID 779328 kern.warning] WARNING: drmach_init: number of logical CPUs (12) in physical processor is not power of 2.
Aug 15 20:09:55 gravity.acheron.be arcsas: [ID 768306 kern.notice]
Aug 15 20:09:55 gravity.acheron.be arcsas2:Areca SAS 6G Host Adapter
Aug 15 20:09:55 gravity.acheron.be arcsas: [ID 768306 kern.notice] arcsas2:ARC13x0 driver v1.00.00.02 2012-03-23
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g1 link up, 1000 Mbps, full duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g3 link up, 1000 Mbps, full duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: aggr1001 link up, 1000 Mbps, full duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1007 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1008 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1005 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1006 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1004 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1003 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:56 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g0 link up, 1000 Mbps, full duplex
Aug 15 20:10:25 gravity.acheron.be unix: [ID 954099 kern.info] NOTICE: IRQ16 is being shared by drivers with different interrupt levels.
Aug 15 20:10:25 gravity.acheron.be This may result in reduced system performance.
Aug 15 20:10:27 gravity.acheron.be pcieb: [ID 586369 kern.info] PCIE-device: pci17d3,1320@0, arcsas2
Aug 15 20:10:27 gravity.acheron.be npe: [ID 236367 kern.info] PCI Express-device: pci17d3,1320@0, arcsas2
Aug 15 20:10:27 gravity.acheron.be genunix: [ID 936769 kern.info] arcsas2 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd20 at arcsas2: target 1 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd20 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@1,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@1,0 (sd20) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 1 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd22 at arcsas2: target 3 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd22 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@3,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@3,0 (sd22) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 3 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd19 at arcsas2: target 0 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd19 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@0,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@0,0 (sd19) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 0 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd21 at arcsas2: target 2 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd21 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@2,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@2,0 (sd21) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 2 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd23 at arcsas2: target 4 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd23 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@4,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@4,0 (sd23) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 4 LUN 0 onlined
Aug 15 20:10:29 gravity.acheron.be arcsas: [ID 768306 kern.notice]
Aug 15 20:10:29 gravity.acheron.be arcsas0:Areca SAS 6G Host Adapter
Aug 15 20:10:29 gravity.acheron.be arcsas: [ID 768306 kern.notice] arcsas0:ARC13x0 driver v1.00.00.02 2012-03-23
Aug 15 20:10:54 gravity.acheron.be unix: [ID 954099 kern.info] NOTICE: IRQ16 is being shared by drivers with different interrupt levels.
Aug 15 20:10:54 gravity.acheron.be This may result in reduced system performance.
Aug 15 20:10:55 gravity.acheron.be svc.startd[10]: [ID 122153 daemon.warning] svc:/system/boot-archive:default: Method or service exit timed out. Killing contract 16.
Aug 15 20:10:56 gravity.acheron.be svc.startd[10]: [ID 636263 daemon.warning] svc:/system/boot-archive:default: Method "/lib/svc/method/boot-archive" failed due to signal KILL.
Aug 15 20:10:56 gravity.acheron.be pcieb: [ID 586369 kern.info] PCIE-device: pci17d3,1320@0, arcsas0
Aug 15 20:10:56 gravity.acheron.be npe: [ID 236367 kern.info] PCI Express-device: pci17d3,1320@0, arcsas0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] arcsas0 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0
Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd17 at arcsas0: target 5 lun 0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd17 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@5,0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@5,0 (sd17) online
Aug 15 20:10:56 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 5 LUN 0 onlined
Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd12 at arcsas0: target 0 lun 0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd12 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@0,0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@0,0 (sd12) online
Aug 15 20:10:56 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 0 LUN 0 onlined
Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd14 at arcsas0: target 2 lun 0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd14 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@2,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@2,0 (sd14) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 2 LUN 0 onlined
Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd15 at arcsas0: target 3 lun 0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd15 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@3,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@3,0 (sd15) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 3 LUN 0 onlined
Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd13 at arcsas0: target 1 lun 0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd13 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@1,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@1,0 (sd13) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 1 LUN 0 onlined
Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd16 at arcsas0: target 4 lun 0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd16 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@4,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@4,0 (sd16) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 4 LUN 0 onlined
Aug 15 20:11:01 gravity.acheron.be genunix: [ID 227219 kern.info] This Solaris instance has UUID 05da9d90-3687-e683-92e2-a79fd35f7b07
Aug 15 20:11:08 gravity.acheron.be genunix: [ID 454863 kern.info] dump on /dev/zvol/dsk/rpool/dump size 16346 MB
Aug 15 20:11:09 gravity.acheron.be rootnex: [ID 349649 kern.info] npe1 at root: space 35 offset 0
Aug 15 20:11:09 gravity.acheron.be genunix: [ID 936769 kern.info] npe1 is /pci@35,0
Aug 15 20:11:09 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pm0
Aug 15 20:11:09 gravity.acheron.be genunix: [ID 936769 kern.info] pm0 is /pseudo/pm@0
Aug 15 20:11:10 gravity.acheron.be rootnex: [ID 349649 kern.info] iscsi0 at root
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] iscsi0 is /iscsi
Aug 15 20:11:10 gravity.acheron.be acpinex: [ID 328922 kern.info] acpinex: sb@0, acpinex1
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] acpinex1 is /fw/sb@0
Aug 15 20:11:10 gravity.acheron.be isa: [ID 202937 kern.info] ISA-device: pit_beep0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] pit_beep0 is /pci@0,0/isa@1f/pit_beep
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: power0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] power0 is /pseudo/power@0
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: srn0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] srn0 is /pseudo/srn@0
Aug 15 20:11:10 gravity.acheron.be /usr/lib/power/powerd: [ID 387247 daemon.error] Able to open /dev/srn
Aug 15 20:11:10 gravity.acheron.be ntpd[300]: [ID 702911 daemon.notice] ntpd 4.2.5p200@1.1948-o Wed Jun 6 13:08:39 BST 2012 (1)
Aug 15 20:11:10 gravity.acheron.be ntpd[332]: [ID 702911 daemon.notice] proto: precision = 0.252 usec
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pseudo1
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] pseudo1 is /pseudo/zconsnex@1
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fct0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] fct0 is /pseudo/fct@0
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: stmf0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] stmf0 is /pseudo/stmf@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: llc10
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] llc10 is /pseudo/llc1@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lofi0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] lofi0 is /pseudo/lofi@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: ramdisk1024
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] ramdisk1024 is /pseudo/ramdisk@1024
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: ucode0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] ucode0 is /pseudo/ucode@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dcpc0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] dcpc0 is /pseudo/dcpc@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dtrace0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] dtrace0 is /pseudo/dtrace@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fasttrap0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fasttrap0 is /pseudo/fasttrap@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fbt0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fbt0 is /pseudo/fbt@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lockstat0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] lockstat0 is /pseudo/lockstat@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: profile0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] profile0 is /pseudo/profile@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: sdt0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] sdt0 is /pseudo/sdt@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: systrace0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] systrace0 is /pseudo/systrace@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fcsm0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fcsm0 is /pseudo/fcsm@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: nvidia255
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] nvidia255 is /pseudo/nvidia@255
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fcp0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fcp0 is /pseudo/fcp@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pool0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] pool0 is /pseudo/pool@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fssnap0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fssnap0 is /pseudo/fssnap@0
Aug 15 20:11:11 gravity.acheron.be ipf: [ID 774698 kern.info] IP Filter: v4.1.9, running.
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: nsmb0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] nsmb0 is /pseudo/nsmb@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: bpf0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] bpf0 is /pseudo/bpf@0
Aug 15 20:11:12 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: kvm0
Aug 15 20:11:12 gravity.acheron.be genunix: [ID 936769 kern.info] kvm0 is /pseudo/kvm@0
Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@0 (zcons0) online
Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@1 (zcons1) online
Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@2 (zcons2) online
Aug 15 20:11:24 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Aug 15 20:11:24 gravity.acheron.be genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Aug 16 04:11:26 gravity.acheron.be smbsrv: [ID 138215 kern.notice] NOTICE: smbd[GRAVITY\htpc]: download share not found
Aug 16 04:11:26 gravity.acheron.be last message repeated 1 time
Aug 16 10:33:08 gravity.acheron.be mDNSResponder: [ID 702911 daemon.error] ERROR: getOptRdata - unknown opt 4
Aug 16 10:34:02 gravity.acheron.be last message repeated 26 times
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dcpc0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] dcpc0 is /pseudo/dcpc@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fasttrap0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] fasttrap0 is /pseudo/fasttrap@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fbt0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] fbt0 is /pseudo/fbt@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lockstat0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] lockstat0 is /pseudo/lockstat@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: profile0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] profile0 is /pseudo/profile@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: sdt0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] sdt0 is /pseudo/sdt@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: systrace0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] systrace0 is /pseudo/systrace@0
This is all I could grab before the system became so unusable I had to disbale CIFS for a few minutes, back to normal now.
I'd love to grab more specific things if it happens again, but I have little experience debugging stuff on solaris.
Updated by Gordon Ross over 4 years ago
Updated by Is this problem still reproducible? If not, let's just close this.
Updated by Jorge Schrauwen over 4 years ago
Hi Gordon,
I no longer have the hardware I had theses issues on.
It went away after I switched to a SuperMicro board + Xeon from an Intel board + i7 combo.
I never did figure out what was causing it aside form disabling CIFS fixed it. (And switched to Samba shortly after)
I guess this can be closed.