Bug #3084
closedSystem slow when accssing many file using CIFS.
0%
Description
General Description
The entire system becomes 'slow' during(possible also after) accessing many files in a large CIFS share.
I am sometimes able to trigger this when for example using 7zip to unpack a large multi part rar archive.
Or when windows media center starts to index a large share for all video/music content.¶
Symptoms
I would generally describe it as system is slow. Although this is not 100% accurate.
Here is an example of running ls in this state, this usually happens on the first attempt, consecutive attempts sometimes work normal or show the same symptom in a lesser degree.¶
@gravity ~ # time ls
scripts TODO
real 0m18.079s
user 0m0.001s
sys 0m0.003s@
More information
tsoome helped me nail it down, below is a bunch of pastes of stuff I looked up for myself or for him.
http://pastebin.com/RUbBP4f4
http://paste.ubuntu.com/1148488/
There were more but I recently clean installed OSX Mountain Lion so I lost them.¶
Some more 'experience' description
Through my poking around and looking at things, I've gotten the feeling that the system is not 'slow' it self. Once for example heavy task gets started... it works as expected without any slowdowns.¶
But there is a huge delay before this task get started.
A nice example is a ssh login in this state can sometimes take 10-20 minutes!
I recently did notice that removing /usr/bin/quota from /etc/profiles, reduces this significantly.
lost test data so this should probably be redone at some point:
Initial assumption was that it was my network setup, I swapped around the nics in the aggr's or tried without aggr. Same effect.¶
Disk latency, local disk intensive tasks have no issue... even in the 'slow' state the work fine once the get started.
Interupts, nothing out of the ordinary
Sysload/idle, again all seemed fine on this front.
Heat issues, happens on both hot days and cold days.
Running for example a stress test tool on a warm then, then booting the system normally, did not trigger the effect.
I also stripped the system of all unneeded components (PCI nics and one of the HBA's, cut down the ram to 16gb. swapping various modules along the way)
Updates all the firmwares and drivers to latest version.
system hardware and configuration
I'll be brief here, most of this is covered in my personal wiki.
Although this is under the personal section:
https://docu.blackdot.be/personal/gravity/planning (hardware)
https://docu.blackdot.be/personal/gravity/notes (configuration)
Login: illumos with password illumos¶
Sorry for not moving it to a public section, I stripped most of my super sensitive information. But I would still like to keep it away from the general public.
The information there is fairly accurate. I did recently switch to zsh but the problem is happening from before this.
Although I run OpenIndiana, OmniOS seems to have the same issue.
I did a clean install (without mirroring the OS), create the user accounts for CIFS and imported my zpools.
I managed to hit the problem with that system too after playing around with windows media center's indexing.
Updated by Jorge Schrauwen almost 10 years ago
- Status changed from New to Feedback
Here is some fresh data from this morning:
-(~)-[✓]-{ time truss ls }-(sjorge@gravity)- execve("/usr/gnu/bin/ls", 0x08047D7C, 0x08047D84) argc = 1 sysinfo(SI_MACHINE, "i86pc", 257) = 6 mmap(0x00000000, 32, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFB0000 mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFA0000 sysconfig(_CONFIG_PAGESIZE) = 4096 mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF90000 mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF80000 memcntl(0xFEFB7000, 32112, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0 memcntl(0x08050000, 21548, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0 resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12 resolvepath("/usr/gnu/bin/ls", "/usr/gnu/bin/ls", 1023) = 15 stat64("/usr/gnu/bin/ls", 0x080479C0) = 0 open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT stat64("/lib/libsec.so.1", 0x08047170) = 0 resolvepath("/lib/libsec.so.1", "/lib/libsec.so.1", 1023) = 16 open("/lib/libsec.so.1", O_RDONLY) = 3 mmapobj(3, MMOBJ_INTERPRET, 0xFEFA0D78, 0x080471DC, 0x00000000) = 0 close(3) = 0 mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF50000 memcntl(0xFEF60000, 15192, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0 stat64("/lib/libc.so.1", 0x08047170) = 0 resolvepath("/lib/libc.so.1", "/lib/libc.so.1", 1023) = 14 open("/lib/libc.so.1", O_RDONLY) = 3 mmapobj(3, MMOBJ_INTERPRET, 0xFEF50550, 0x080471DC, 0x00000000) = 0 close(3) = 0 memcntl(0xFEE10000, 179176, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0 mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEE00000 mmap(0x00010000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEDF0000 getcontext(0x08047820) getrlimit(RLIMIT_STACK, 0x08047818) = 0 getpid() = 3756 [3755] lwp_private(0, 1, 0xFEDF2A40) = 0x000001C3 setustack(0xFEDF2AA0) sysi86(SI86FPSTART, 0xFEF49C24, 0x0000133F, 0x00001F80) = 0x00000001 open("/usr/lib/locale//en_US.UTF-8/LC_CTYPE/LCL_DATA", O_RDONLY) = 3 fstat64(3, 0x080475B0) = 0 fstat(3, 0x08047710) = 0 brk(0x08083910) = 0 brk(0x0808B910) = 0 llseek(3, 0, SEEK_CUR) = 0 llseek(3, 0, SEEK_SET) = 0 fstat64(3, 0x080475A0) = 0 brk(0x0808B910) = 0 brk(0x08093910) = 0 fstat64(3, 0x080474B0) = 0 ioctl(3, TCGETA, 0x08047550) Err#25 ENOTTY read(3, " R u n e M a g 1 U T F -".., 30208) = 30188 brk(0x08093910) = 0 brk(0x0809D910) = 0 llseek(3, 0, SEEK_CUR) = 30188 close(3) = 0 open("/usr/lib/locale//en_US.UTF-8/LC_NUMERIC/LCL_DATA", O_RDONLY) = 3 fstat64(3, 0x08047590) = 0 fstat(3, 0x08047B60) = 0 read(3, " .\n ,\n 3\n", 6) = 6 close(3) = 0 open("/usr/lib/locale//en_US.UTF-8/LC_TIME/LCL_DATA", O_RDONLY) = 3 fstat64(3, 0x080475A0) = 0 fstat(3, 0x08047B70) = 0 read(3, " J a n\n F e b\n M a r\n".., 308) = 308 close(3) = 0 open("/usr/lib/locale//en_US.UTF-8/LC_COLLATE/LCL_DATA", O_RDONLY) = 3 fstat64(3, 0x080475C0) = 0 fstat(3, 0x08047BA0) = 0 mmap(0x00000000, 75408, PROT_READ, MAP_PRIVATE, 3, 0) = 0xFEDDC000 close(3) = 0 open("/usr/lib/locale//en_US.UTF-8/LC_MONETARY/LCL_DATA", O_RDONLY) = 3 fstat64(3, 0x08047590) = 0 fstat(3, 0x08047B60) = 0 read(3, " U S D \n $\n .\n ,\n 3".., 44) = 44 close(3) = 0 open("/usr/lib/locale//en_US.UTF-8/LC_MESSAGES/LCL_DATA", O_RDONLY) = 3 fstat64(3, 0x080475A0) = 0 fstat(3, 0x08047B70) = 0 read(3, " ^ ( ( [ y Y ] ( [ e E ]".., 45) = 45 close(3) = 0 ioctl(1, TCGETA, 0x08047C50) = 0 ioctl(1, TIOCGWINSZ, 0x08047CB8) = 0 open(".", O_RDONLY|O_NDELAY|O_LARGEFILE) = 3 fstat64(3, 0x08047940) = 0 fcntl(3, F_SETFD, 0x00000001) = 0 fstat64(3, 0x08047A40) = 0 getdents64(3, 0xFEDF4000, 8192) = 528 getdents64(3, 0xFEDF4000, 8192) = 0 close(3) = 0 close(1) = 0 close(2) = 0 _exit(0) truss ls 0.00s user 0.00s system 0% cpu 26.226 total
-(~)-[✓]-{ vmstat 5 5 }-(sjorge@gravity)- kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s0 s3 s9 s1 in sy cs us sy id 1 0 0 34207128 21780212 7 34 0 0 0 0 26 3 -0 1 1 1436 868 1100 0 0 100 0 0 0 30331980 17890896 7 14 0 0 0 0 0 0 0 4 4 1269 375 949 0 0 100 1 0 0 30331868 17890816 0 0 0 0 0 0 0 0 0 0 0 1087 368 814 0 0 100 1 0 0 30331868 17890816 0 0 0 0 0 0 0 0 0 0 0 1092 405 824 0 0 100 0 0 0 30331868 17890816 0 0 0 0 0 0 0 0 0 0 0 1141 354 808 0 0 100
-(~)-[✓]-{ mpstat 5 5 }-(sjorge@gravity)- CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 2 0 3 455 219 85 0 2 4 0 61 0 0 0 99 1 5 0 8 113 33 134 0 2 4 0 103 0 0 0 99 2 5 0 5 96 25 110 0 2 3 0 95 0 0 0 99 3 2 0 3 79 23 85 0 1 3 0 47 0 0 0 100 4 2 0 2 61 14 64 0 1 3 0 67 0 0 0 100 5 3 0 5 86 21 100 0 2 3 0 83 0 0 0 100 6 4 0 5 96 25 109 0 2 3 0 82 0 0 0 100 7 1 0 2 101 53 67 0 1 5 0 49 0 0 0 100 8 1 0 2 60 15 61 0 1 2 0 39 0 0 0 100 9 4 0 5 100 31 109 0 2 4 0 109 0 0 0 100 10 5 0 6 104 35 109 0 2 4 0 90 0 0 0 100 11 1 0 2 83 36 65 0 1 3 0 37 0 0 0 100 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 333 110 56 0 2 1 0 17 0 0 0 100 1 0 0 7 85 20 96 0 3 4 0 40 0 0 0 100 2 0 0 0 85 30 90 0 3 6 0 46 0 1 0 99 3 0 0 9 53 29 34 0 2 3 0 17 0 0 0 100 4 0 0 0 86 32 94 0 1 2 0 30 0 0 0 100 5 0 0 9 127 55 136 0 2 5 0 16 0 0 0 100 6 1 0 0 67 5 59 0 3 2 0 71 0 0 0 100 7 0 0 0 89 27 77 0 1 2 0 89 0 0 0 100 8 0 0 2 96 27 83 0 1 2 0 27 0 0 0 100 9 0 0 9 60 26 89 0 1 4 0 90 0 0 0 100 10 0 0 2 65 16 60 0 1 4 0 41 0 0 0 100 11 0 0 0 51 11 38 0 1 1 0 10 0 0 0 100 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 315 105 24 0 0 0 0 6 0 0 0 100 1 0 0 0 49 9 41 0 1 0 0 23 0 0 0 100 2 0 0 0 53 12 47 0 1 0 0 16 0 1 0 99 3 0 0 0 42 9 40 0 1 0 0 8 0 0 0 100 4 0 0 0 63 19 60 0 2 0 0 5 0 0 0 100 5 0 0 4 125 54 121 0 1 0 0 9 0 0 0 100 6 1 0 4 58 12 59 0 1 0 0 89 0 0 0 100 7 0 0 0 35 3 25 0 1 0 0 11 0 0 0 100 8 0 0 0 130 48 119 0 0 0 0 47 0 0 0 100 9 0 0 4 96 42 103 0 1 0 0 60 0 0 0 100 10 0 0 4 129 49 135 0 1 1 0 83 0 0 0 100 11 0 0 0 32 6 19 0 0 0 0 3 0 0 0 100 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 4 331 105 72 0 2 0 0 20 0 0 0 100 1 141 0 9 74 19 77 0 3 1 0 110 0 0 0 100 2 0 0 12 94 31 96 0 2 0 0 38 0 0 0 99 3 0 0 0 66 16 58 0 2 1 0 22 0 0 0 100 4 0 0 11 51 7 43 0 1 1 0 11 0 1 0 99 5 69 0 11 93 35 81 0 1 1 0 35 0 0 0 100 6 80 0 4 48 4 32 0 1 1 0 103 0 0 0 100 7 0 0 0 28 2 14 0 0 0 0 4 0 0 0 100 8 0 0 18 124 50 111 0 0 0 0 49 0 0 0 100 9 0 0 9 91 35 78 0 1 0 0 27 0 0 0 100 10 232 0 18 126 42 128 0 1 1 0 798 1 0 0 99 11 0 0 0 39 5 20 0 1 0 0 4 0 0 0 100 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 0 0 0 331 105 45 0 1 2 0 12 0 0 0 100 1 0 0 7 92 26 110 0 1 3 0 66 0 0 0 100 2 0 0 4 38 7 38 0 2 3 0 9 0 1 0 99 3 0 0 13 76 34 58 0 1 1 0 34 0 0 0 100 4 0 0 0 47 3 37 0 1 1 0 5 0 0 0 100 5 0 0 4 63 17 62 0 2 2 0 15 0 0 0 100 6 0 0 0 72 16 69 0 3 2 0 89 0 0 0 100 7 0 0 11 46 10 42 0 2 3 0 16 0 0 0 100 8 0 0 13 152 59 141 0 2 1 0 52 0 0 0 100 9 0 0 18 208 95 200 0 1 1 0 47 0 0 0 100 10 0 0 4 81 15 65 0 1 3 0 10 0 0 0 100 11 0 0 0 28 3 15 0 1 1 0 4 0 0 0 100
-(~)-[✗]-{ intrstat }-(root@gravity)- device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim -------------+------------------------------------------------------------------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.2 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#3 | 1 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 147 0.2 device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim -------------+------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 1 0.0 e1000g#1 | 0 0.0 15 0.0 0 0.0 0 0.0 e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 12 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim -------------+------------------------------------------------------------------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 2 0.0 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#3 | 2 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 2 0.0 device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim -------------+------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 2 0.0 e1000g#1 | 0 0.0 22 0.0 0 0.0 0 0.0 e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 16 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 ehci#1 | 1 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim -------------+------------------------------------------------------------------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#3 | 1 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim -------------+------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 1 0.0 e1000g#1 | 0 0.0 18 0.0 0 0.0 0 0.0 e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 16 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 device | cpu0 %tim cpu1 %tim cpu2 %tim cpu3 %tim cpu4 %tim cpu5 %tim cpu6 %tim cpu7 %tim -------------+------------------------------------------------------------------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 88 0.1 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#3 | 1 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 0 0.0 1 0.0 device | cpu8 %tim cpu9 %tim cpu10 %tim cpu11 %tim -------------+------------------------------------------------------------ ahci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#0 | 0 0.0 0 0.0 0 0.0 0 0.0 arcsas#2 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#0 | 0 0.0 0 0.0 0 0.0 1 0.0 e1000g#1 | 0 0.0 21 0.0 0 0.0 0 0.0 e1000g#3 | 0 0.0 0 0.0 0 0.0 0 0.0 e1000g#4 | 0 0.0 0 0.0 15 0.0 0 0.0 ehci#0 | 0 0.0 0 0.0 0 0.0 0 0.0 ehci#1 | 0 0.0 0 0.0 0 0.0 0 0.0 nvidia#1 | 0 0.0 0 0.0 0 0.0 0 0.0
-(DTT)-[✗]-{ dtrace -n 'syscall:::entry { @num[probefunc] = count(); }' }-(root@gravity)- dtrace: description 'syscall:::entry ' matched 231 probes ^X^C getloadavg 1 kill 1 lwp_detach 1 lwp_mutex_timedlock 1 lwp_mutex_unlock 1 mmap 1 setsockopt 1 shutdown 1 sigpending 1 sendto 2 umount2 2 statvfs64 3 sysconfig 3 alarm 4 fstat 4 getsockname 4 pause 4 recvfrom 4 sigaction 4 openat 6 putmsg 6 rename 6 brk 8 accept 11 getdents64 12 lwp_continue 13 lwp_create 13 lwp_exit 13 uucopy 13 pread 14 schedctl 14 clock_gettime 16 lwp_kill 17 getpeername 18 unlink 18 send 23 c2audit 25 portfs 25 getuid 36 writev 37 getsockopt 40 connect 42 so_socket 49 open 62 stat 71 fdsync 72 open64 72 getmsg 75 stat64 76 sendmsg 92 recvmsg 95 pset 141 pwrite64 180 close 225 waitsys 237 sigsuspend 238 p_online 256 setcontext 264 nanosleep 276 pread64 503 access 960 lwp_sigmask 960 read 1103 write 1177 llseek 1210 fstat64 1546 doorfs 1614 getpid 1667 fcntl 2304 gtime 2422 recv 5302 lwp_park 23651 pollsys 44133 ioctl 64358
(DTT)-[✓]-{ ./hotkernel }-(root@gravity)- Sampling... Hit Ctrl-C to end. ^C FUNCTION COUNT PCNT unix`page_unlock 1 0.0% genunix`thread_free_prevent 1 0.0% aggr`aggr_find_tx_ring 1 0.0% unix`gdt_ucode_model 1 0.0% arcsas`arcsas_build_os_ccb 1 0.0% genunix`new_mstate 1 0.0% genunix`clock_tick 1 0.0% SDC`sysdc_wakeup 1 0.0% genunix`callout_list_get 1 0.0% unix`xsave_ctxt 1 0.0% unix`reset_turbo_info 1 0.0% zfs`zio_vdev_io_done 1 0.0% genunix`taskq_ent_alloc 1 0.0% sha1`sha1_block_data_order 1 0.0% sd`sdstrategy 1 0.0% genunix`kmem_cache_alloc 1 0.0% unix`set_freemem 1 0.0% unix`cpupm_wakeup_cstate_data 1 0.0% genunix`pcache_poll 1 0.0% genunix`callout_list_expire 1 0.0% genunix`lwp_hash_lookup_and_lock 1 0.0% unix`rw_exit 1 0.0% stmf`stmf_svc_timeout 1 0.0% arcsas`arcsas_device_get_sent_ccb_from_list 1 0.0% zfs`vdev_queue_io_remove 1 0.0% rootnex`rootnex_coredma_allochdl 1 0.0% genunix`copyin_args32 1 0.0% genunix`callout_downheap 1 0.0% zfs`vdev_queue_offset_compare 1 0.0% genunix`setrun_locked 1 0.0% unix`strcpy 1 0.0% unix`cpu_resched 1 0.0% unix`_sys_sysenter_post_swapgs 1 0.0% unix`bcopy 1 0.0% genunix`disp_lock_exit_high 1 0.0% genunix`avl_walk 1 0.0% unix`av_dispatch_softvect 1 0.0% zfs`dmu_object_size_from_db 1 0.0% TS`ts_sleep 1 0.0% genunix`mexchange 1 0.0% arcsas`arcsas_ccb_done 1 0.0% genunix`callout_expire 1 0.0% ip`squeue_worker 1 0.0% arcsas`arcsas_check_irq_state 1 0.0% zfs`buf_hash_find 1 0.0% unix`sep_restore 1 0.0% unix`prunstop 1 0.0% sd`sdintr 1 0.0% unix`xc_call 1 0.0% zfs`arc_buf_remove_ref 1 0.0% genunix`syscall_exit 1 0.0% zfs`space_map_seg_compare 1 0.0% genunix`cyclic_reprogram_cyclic 1 0.0% genunix`taskq_thread 1 0.0% pcplusmp`apic_unset_idlecpu 1 0.0% genunix`cv_waituntil_sig 1 0.0% genunix`schedctl_set_cidpri 1 0.0% unix`x86_hm_exit 1 0.0% genunix`cv_wait 1 0.0% genunix`fd_find 1 0.0% unix`cpupm_utilization_event 1 0.0% zfs`dbuf_hash 1 0.0% pcplusmp`apic_send_ipi 1 0.0% unix`cpupm_state_change 1 0.0% genunix`schedctl_restore 1 0.0% genunix`ddi_dma_sync 1 0.0% unix`atomic_add_64_nv 1 0.0% genunix`list_remove 1 0.0% zfs`dnode_block_freed 1 0.0% genunix`avl_numnodes 1 0.0% unix`hat_kpm_pfn2va 1 0.0% genunix`sleepq_insert 1 0.0% genunix`expire_tick 1 0.0% unix`idle_exit 1 0.0% unix`av_clear_softint_pending 1 0.0% genunix`cv_wait_sig_swap_core 1 0.0% unix`rdmsr 1 0.0% genunix`turnstile_block 1 0.0% genunix`cv_unsleep 1 0.0% genunix`sys_nullsys 1 0.0% unix`clock_tick_process 1 0.0% arcsas`arcsas_set_lba_and_sector_count 1 0.0% unix`atomic_add_64 1 0.0% unix`atomic_add_32_nv 1 0.0% unix`rw_enter 2 0.0% genunix`poll_common 2 0.0% unix`gdt_update_usegd 2 0.0% genunix`cpu_update_pct 2 0.0% unix`hat_switch 2 0.0% unix`lock_try 2 0.0% unix`wrmsr 2 0.0% genunix`cyclic_softint 2 0.0% unix`sep_save 2 0.0% unix`atomic_cas_32 2 0.0% unix`setbackdq 2 0.0% genunix`untimeout_default 2 0.0% genunix`thread_lock 2 0.0% unix`cpu_acpi_idle 2 0.0% unix`bitset_atomic_add 2 0.0% unix`cpupm_next_cstate 2 0.0% unix`_resume_from_idle 2 0.0% genunix`ddi_get_lbolt 2 0.0% unix`clock_tick_execute_common 2 0.0% unix`lwp_segregs_restore32 2 0.0% genunix`cv_timedwait_sig_hires 2 0.0% unix`lock_set_spl 2 0.0% genunix`taskq_dispatch 2 0.0% mac`mac_rx_deliver 2 0.0% genunix`callout_heap_delete 3 0.0% unix`mutex_delay_default 3 0.0% unix`mul32 3 0.0% zfs`l2arc_write_buffers 3 0.0% unix`0xfffffffffb85 3 0.0% unix`disp_anywork 3 0.0% unix`mutex_tryenter 3 0.0% genunix`clock 3 0.0% unix`default_lock_delay 3 0.0% unix`tsc_scalehrtime 3 0.0% unix`cmt_ev_thread_swtch 3 0.0% unix`atomic_and_64 3 0.0% unix`splr 3 0.0% unix`disp_ratify 3 0.0% genunix`syscall_entry 3 0.0% genunix`timeout_generic 3 0.0% dtrace`dtrace_dynvar_clean 3 0.0% rootnex`rootnex_teardown_copybuf 3 0.0% unix`bzero 3 0.0% genunix`post_syscall 4 0.0% unix`cmt_balance 4 0.0% unix`mutex_owned 4 0.0% unix`mutex_exit 4 0.0% unix`page_pptonum 4 0.0% unix`lock_set 4 0.0% unix`idle 4 0.0% unix`bitset_in_set 4 0.0% sha2`SHA256TransformBlocks 5 0.0% unix`swtch 5 0.0% unix`resume 5 0.0% unix`dispatch_hilevel 5 0.0% unix`ddi_get32 5 0.0% unix`membar_enter 5 0.0% unix`prefetch_page_r 6 0.0% genunix`syscall_mstate 6 0.0% genunix`restorectx 6 0.0% unix`av_check_softint_pending 7 0.0% unix`atomic_add_32 8 0.0% zfs`lzjb_compress 9 0.0% unix`atomic_or_64 9 0.0% unix`tsc_gethrtime_delta 10 0.0% zfs`l2arc_write_eligible 10 0.0% genunix`lbolt_event_driven 11 0.0% unix`mutex_enter 13 0.0% genunix`list_next 14 0.0% unix`disp_getwork 18 0.0% unix`tsc_gethrtimeunscaled_delta 19 0.0% unix`dispatch_softint 26 0.0% zfs`buf_hash 27 0.0% unix`do_splx 47 0.0% unix`page_nextn 103 0.0% genunix`fsflush_do_pages 108 0.0% unix`i86_mwait 601 0.1% unix`acpi_cpu_cstate 405605 99.7%
-(~adm)-[✓]-{ tail -n200 messages }-(root@gravity)- Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hubd1 is /pci@0,0/pci8086,4953@1d/hub@1 Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1d/hub@1 (hubd1) online Aug 15 20:09:47 gravity.acheron.be drmach_acpi: [ID 779328 kern.warning] WARNING: drmach_init: number of logical CPUs (12) in physical processor is not power of 2. Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 device (usb4f3,103) operating at low speed (USB 1.x) on USB 2.0 external hub: device@5, usb_mid1 at bus address 3 Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] usb_mid1 is /pci@0,0/pci8086,4953@1a/hub@1/device@5 Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5 (usb_mid1) online Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 interface (usbif4f3,103.config1.0) operating at low speed (USB 1.x) on USB 2.0 external hub: keyboard@0, hid4 at bus address 3 Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hid4 is /pci@0,0/pci8086,4953@1a/hub@1/device@5/keyboard@0 Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5/keyboard@0 (hid4) online Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 interface (usbif4f3,103.config1.1) operating at low speed (USB 1.x) on USB 2.0 external hub: input@1, hid5 at bus address 3 Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hid5 is /pci@0,0/pci8086,4953@1a/hub@1/device@5/input@1 Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5/input@1 (hid5) online Aug 15 20:09:48 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: audio0 Aug 15 20:09:48 gravity.acheron.be genunix: [ID 936769 kern.info] audio0 is /pseudo/audio@0 Aug 15 20:09:51 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10d3 (e1000g) instance 1 irq 0x34 vector 0x60 ioapic 0xff intin 0xff is bound to cpu 9 Aug 15 20:09:51 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g1 registered Aug 15 20:09:51 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,107d (e1000g) instance 4 irq 0x35 vector 0x61 ioapic 0xff intin 0xff is bound to cpu 10 Aug 15 20:09:51 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g4 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: aggr1000 registered Aug 15 20:09:52 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pci8086,1502 (e1000g) instance 0 irq 0x36 vector 0x62 ioapic 0xff intin 0xff is bound to cpu 11 Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g0 registered Aug 15 20:09:52 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10d3 (e1000g) instance 3 irq 0x37 vector 0x63 ioapic 0xff intin 0xff is bound to cpu 0 Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g3 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: aggr1001 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1002 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1003 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1002 link up, 0 Mbps, unknown duplex Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1004 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1005 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1006 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1007 registered Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1008 registered Aug 15 20:09:53 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: devinfo0 Aug 15 20:09:53 gravity.acheron.be genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0 Aug 15 20:09:54 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: zfs0 Aug 15 20:09:54 gravity.acheron.be genunix: [ID 936769 kern.info] zfs0 is /pseudo/zfs@0 Aug 15 20:09:54 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g4 link up, 1000 Mbps, full duplex Aug 15 20:09:54 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: aggr1000 link up, 1000 Mbps, full duplex Aug 15 20:09:54 gravity.acheron.be drmach_acpi: [ID 779328 kern.warning] WARNING: drmach_init: number of logical CPUs (12) in physical processor is not power of 2. Aug 15 20:09:55 gravity.acheron.be arcsas: [ID 768306 kern.notice] Aug 15 20:09:55 gravity.acheron.be arcsas2:Areca SAS 6G Host Adapter Aug 15 20:09:55 gravity.acheron.be arcsas: [ID 768306 kern.notice] arcsas2:ARC13x0 driver v1.00.00.02 2012-03-23 Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g1 link up, 1000 Mbps, full duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g3 link up, 1000 Mbps, full duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: aggr1001 link up, 1000 Mbps, full duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1007 link up, 1000 Mbps, unknown duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1008 link up, 1000 Mbps, unknown duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1005 link up, 1000 Mbps, unknown duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1006 link up, 1000 Mbps, unknown duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1004 link up, 1000 Mbps, unknown duplex Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1003 link up, 1000 Mbps, unknown duplex Aug 15 20:09:56 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g0 link up, 1000 Mbps, full duplex Aug 15 20:10:25 gravity.acheron.be unix: [ID 954099 kern.info] NOTICE: IRQ16 is being shared by drivers with different interrupt levels. Aug 15 20:10:25 gravity.acheron.be This may result in reduced system performance. Aug 15 20:10:27 gravity.acheron.be pcieb: [ID 586369 kern.info] PCIE-device: pci17d3,1320@0, arcsas2 Aug 15 20:10:27 gravity.acheron.be npe: [ID 236367 kern.info] PCI Express-device: pci17d3,1320@0, arcsas2 Aug 15 20:10:27 gravity.acheron.be genunix: [ID 936769 kern.info] arcsas2 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0 Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd20 at arcsas2: target 1 lun 0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd20 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@1,0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@1,0 (sd20) online Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 1 LUN 0 onlined Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd22 at arcsas2: target 3 lun 0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd22 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@3,0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@3,0 (sd22) online Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 3 LUN 0 onlined Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd19 at arcsas2: target 0 lun 0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd19 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@0,0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@0,0 (sd19) online Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 0 LUN 0 onlined Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd21 at arcsas2: target 2 lun 0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd21 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@2,0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@2,0 (sd21) online Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 2 LUN 0 onlined Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd23 at arcsas2: target 4 lun 0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd23 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@4,0 Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@4,0 (sd23) online Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 4 LUN 0 onlined Aug 15 20:10:29 gravity.acheron.be arcsas: [ID 768306 kern.notice] Aug 15 20:10:29 gravity.acheron.be arcsas0:Areca SAS 6G Host Adapter Aug 15 20:10:29 gravity.acheron.be arcsas: [ID 768306 kern.notice] arcsas0:ARC13x0 driver v1.00.00.02 2012-03-23 Aug 15 20:10:54 gravity.acheron.be unix: [ID 954099 kern.info] NOTICE: IRQ16 is being shared by drivers with different interrupt levels. Aug 15 20:10:54 gravity.acheron.be This may result in reduced system performance. Aug 15 20:10:55 gravity.acheron.be svc.startd[10]: [ID 122153 daemon.warning] svc:/system/boot-archive:default: Method or service exit timed out. Killing contract 16. Aug 15 20:10:56 gravity.acheron.be svc.startd[10]: [ID 636263 daemon.warning] svc:/system/boot-archive:default: Method "/lib/svc/method/boot-archive" failed due to signal KILL. Aug 15 20:10:56 gravity.acheron.be pcieb: [ID 586369 kern.info] PCIE-device: pci17d3,1320@0, arcsas0 Aug 15 20:10:56 gravity.acheron.be npe: [ID 236367 kern.info] PCI Express-device: pci17d3,1320@0, arcsas0 Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] arcsas0 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0 Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd17 at arcsas0: target 5 lun 0 Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd17 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@5,0 Aug 15 20:10:56 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@5,0 (sd17) online Aug 15 20:10:56 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 5 LUN 0 onlined Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd12 at arcsas0: target 0 lun 0 Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd12 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@0,0 Aug 15 20:10:56 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@0,0 (sd12) online Aug 15 20:10:56 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 0 LUN 0 onlined Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd14 at arcsas0: target 2 lun 0 Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd14 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@2,0 Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@2,0 (sd14) online Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 2 LUN 0 onlined Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd15 at arcsas0: target 3 lun 0 Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd15 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@3,0 Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@3,0 (sd15) online Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 3 LUN 0 onlined Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd13 at arcsas0: target 1 lun 0 Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd13 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@1,0 Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@1,0 (sd13) online Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 1 LUN 0 onlined Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd16 at arcsas0: target 4 lun 0 Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd16 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@4,0 Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@4,0 (sd16) online Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 4 LUN 0 onlined Aug 15 20:11:01 gravity.acheron.be genunix: [ID 227219 kern.info] This Solaris instance has UUID 05da9d90-3687-e683-92e2-a79fd35f7b07 Aug 15 20:11:08 gravity.acheron.be genunix: [ID 454863 kern.info] dump on /dev/zvol/dsk/rpool/dump size 16346 MB Aug 15 20:11:09 gravity.acheron.be rootnex: [ID 349649 kern.info] npe1 at root: space 35 offset 0 Aug 15 20:11:09 gravity.acheron.be genunix: [ID 936769 kern.info] npe1 is /pci@35,0 Aug 15 20:11:09 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pm0 Aug 15 20:11:09 gravity.acheron.be genunix: [ID 936769 kern.info] pm0 is /pseudo/pm@0 Aug 15 20:11:10 gravity.acheron.be rootnex: [ID 349649 kern.info] iscsi0 at root Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] iscsi0 is /iscsi Aug 15 20:11:10 gravity.acheron.be acpinex: [ID 328922 kern.info] acpinex: sb@0, acpinex1 Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] acpinex1 is /fw/sb@0 Aug 15 20:11:10 gravity.acheron.be isa: [ID 202937 kern.info] ISA-device: pit_beep0 Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] pit_beep0 is /pci@0,0/isa@1f/pit_beep Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: power0 Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] power0 is /pseudo/power@0 Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: srn0 Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] srn0 is /pseudo/srn@0 Aug 15 20:11:10 gravity.acheron.be /usr/lib/power/powerd: [ID 387247 daemon.error] Able to open /dev/srn Aug 15 20:11:10 gravity.acheron.be ntpd[300]: [ID 702911 daemon.notice] ntpd 4.2.5p200@1.1948-o Wed Jun 6 13:08:39 BST 2012 (1) Aug 15 20:11:10 gravity.acheron.be ntpd[332]: [ID 702911 daemon.notice] proto: precision = 0.252 usec Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pseudo1 Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] pseudo1 is /pseudo/zconsnex@1 Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fct0 Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] fct0 is /pseudo/fct@0 Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: stmf0 Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] stmf0 is /pseudo/stmf@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: llc10 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] llc10 is /pseudo/llc1@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lofi0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] lofi0 is /pseudo/lofi@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: ramdisk1024 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] ramdisk1024 is /pseudo/ramdisk@1024 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: ucode0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] ucode0 is /pseudo/ucode@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dcpc0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] dcpc0 is /pseudo/dcpc@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dtrace0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] dtrace0 is /pseudo/dtrace@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fasttrap0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fasttrap0 is /pseudo/fasttrap@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fbt0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fbt0 is /pseudo/fbt@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lockstat0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] lockstat0 is /pseudo/lockstat@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: profile0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] profile0 is /pseudo/profile@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: sdt0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] sdt0 is /pseudo/sdt@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: systrace0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] systrace0 is /pseudo/systrace@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fcsm0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fcsm0 is /pseudo/fcsm@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: nvidia255 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] nvidia255 is /pseudo/nvidia@255 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fcp0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fcp0 is /pseudo/fcp@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pool0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] pool0 is /pseudo/pool@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fssnap0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fssnap0 is /pseudo/fssnap@0 Aug 15 20:11:11 gravity.acheron.be ipf: [ID 774698 kern.info] IP Filter: v4.1.9, running. Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: nsmb0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] nsmb0 is /pseudo/nsmb@0 Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: bpf0 Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] bpf0 is /pseudo/bpf@0 Aug 15 20:11:12 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: kvm0 Aug 15 20:11:12 gravity.acheron.be genunix: [ID 936769 kern.info] kvm0 is /pseudo/kvm@0 Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@0 (zcons0) online Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@1 (zcons1) online Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@2 (zcons2) online Aug 15 20:11:24 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: devinfo0 Aug 15 20:11:24 gravity.acheron.be genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0 Aug 16 04:11:26 gravity.acheron.be smbsrv: [ID 138215 kern.notice] NOTICE: smbd[GRAVITY\htpc]: download share not found Aug 16 04:11:26 gravity.acheron.be last message repeated 1 time Aug 16 10:33:08 gravity.acheron.be mDNSResponder: [ID 702911 daemon.error] ERROR: getOptRdata - unknown opt 4 Aug 16 10:34:02 gravity.acheron.be last message repeated 26 times Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dcpc0 Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] dcpc0 is /pseudo/dcpc@0 Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fasttrap0 Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] fasttrap0 is /pseudo/fasttrap@0 Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fbt0 Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] fbt0 is /pseudo/fbt@0 Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lockstat0 Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] lockstat0 is /pseudo/lockstat@0 Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: profile0 Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] profile0 is /pseudo/profile@0 Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: sdt0 Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] sdt0 is /pseudo/sdt@0 Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: systrace0 Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] systrace0 is /pseudo/systrace@0
This is all I could grab before the system became so unusable I had to disbale CIFS for a few minutes, back to normal now.
I'd love to grab more specific things if it happens again, but I have little experience debugging stuff on solaris.
Updated by Gordon Ross almost 7 years ago
Is this problem still reproducible? If not, let's just close this.
Updated by Jorge Schrauwen almost 7 years ago
Hi Gordon,
I no longer have the hardware I had theses issues on.
It went away after I switched to a SuperMicro board + Xeon from an Intel board + i7 combo.
I never did figure out what was causing it aside form disabling CIFS fixed it. (And switched to Samba shortly after)
I guess this can be closed.