Project

General

Profile

Bug #3084

System slow when accssing many file using CIFS.

Added by Jorge Schrauwen almost 8 years ago. Updated over 4 years ago.

Status:
Feedback
Priority:
Low
Assignee:
-
Category:
cifs - CIFS server and client
Start date:
2012-08-15
Due date:
% Done:

0%

Estimated time:
Difficulty:
Hard
Tags:
needs-triage
Gerrit CR:

Description

General Description
The entire system becomes 'slow' during(possible also after) accessing many files in a large CIFS share.
I am sometimes able to trigger this when for example using 7zip to unpack a large multi part rar archive.
Or when windows media center starts to index a large share for all video/music content.

Symptoms
I would generally describe it as system is slow. Although this is not 100% accurate.
Here is an example of running ls in this state, this usually happens on the first attempt, consecutive attempts sometimes work normal or show the same symptom in a lesser degree.

@gravity ~ # time ls
scripts TODO

real 0m18.079s
user 0m0.001s
sys 0m0.003s@

More information
tsoome helped me nail it down, below is a bunch of pastes of stuff I looked up for myself or for him.
http://pastebin.com/RUbBP4f4
http://paste.ubuntu.com/1148488/
There were more but I recently clean installed OSX Mountain Lion so I lost them.

Some more 'experience' description
Through my poking around and looking at things, I've gotten the feeling that the system is not 'slow' it self. Once for example heavy task gets started... it works as expected without any slowdowns.

But there is a huge delay before this task get started.
A nice example is a ssh login in this state can sometimes take 10-20 minutes!
I recently did notice that removing /usr/bin/quota from /etc/profiles, reduces this significantly.

lost test data so this should probably be redone at some point:
Initial assumption was that it was my network setup, I swapped around the nics in the aggr's or tried without aggr. Same effect.

Disk latency, local disk intensive tasks have no issue... even in the 'slow' state the work fine once the get started.

Interupts, nothing out of the ordinary

Sysload/idle, again all seemed fine on this front.

Heat issues, happens on both hot days and cold days.
Running for example a stress test tool on a warm then, then booting the system normally, did not trigger the effect.
I also stripped the system of all unneeded components (PCI nics and one of the HBA's, cut down the ram to 16gb. swapping various modules along the way)

Updates all the firmwares and drivers to latest version.

system hardware and configuration
I'll be brief here, most of this is covered in my personal wiki.
Although this is under the personal section:
https://docu.blackdot.be/personal/gravity/planning (hardware)
https://docu.blackdot.be/personal/gravity/notes (configuration)
Login: illumos with password illumos

Sorry for not moving it to a public section, I stripped most of my super sensitive information. But I would still like to keep it away from the general public.

The information there is fairly accurate. I did recently switch to zsh but the problem is happening from before this.

Although I run OpenIndiana, OmniOS seems to have the same issue.
I did a clean install (without mirroring the OS), create the user accounts for CIFS and imported my zpools.
I managed to hit the problem with that system too after playing around with windows media center's indexing.

History

#1

Updated by Jorge Schrauwen almost 8 years ago

  • Status changed from New to Feedback

Here is some fresh data from this morning:

-(~)-[✓]-{ time truss ls                                                                                              }-(sjorge@gravity)-
execve("/usr/gnu/bin/ls", 0x08047D7C, 0x08047D84)  argc = 1
sysinfo(SI_MACHINE, "i86pc", 257)        = 6
mmap(0x00000000, 32, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFB0000
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFA0000
sysconfig(_CONFIG_PAGESIZE)            = 4096
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF90000
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF80000
memcntl(0xFEFB7000, 32112, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
memcntl(0x08050000, 21548, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12
resolvepath("/usr/gnu/bin/ls", "/usr/gnu/bin/ls", 1023) = 15
stat64("/usr/gnu/bin/ls", 0x080479C0)        = 0
open("/var/ld/ld.config", O_RDONLY)        Err#2 ENOENT
stat64("/lib/libsec.so.1", 0x08047170)        = 0
resolvepath("/lib/libsec.so.1", "/lib/libsec.so.1", 1023) = 16
open("/lib/libsec.so.1", O_RDONLY)        = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFEFA0D78, 0x080471DC, 0x00000000) = 0
close(3)                    = 0
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEF50000
memcntl(0xFEF60000, 15192, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
stat64("/lib/libc.so.1", 0x08047170)        = 0
resolvepath("/lib/libc.so.1", "/lib/libc.so.1", 1023) = 14
open("/lib/libc.so.1", O_RDONLY)        = 3
mmapobj(3, MMOBJ_INTERPRET, 0xFEF50550, 0x080471DC, 0x00000000) = 0
close(3)                    = 0
memcntl(0xFEE10000, 179176, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEE00000
mmap(0x00010000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEDF0000
getcontext(0x08047820)
getrlimit(RLIMIT_STACK, 0x08047818)        = 0
getpid()                    = 3756 [3755]
lwp_private(0, 1, 0xFEDF2A40)            = 0x000001C3
setustack(0xFEDF2AA0)
sysi86(SI86FPSTART, 0xFEF49C24, 0x0000133F, 0x00001F80) = 0x00000001
open("/usr/lib/locale//en_US.UTF-8/LC_CTYPE/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475B0)                = 0
fstat(3, 0x08047710)                = 0
brk(0x08083910)                    = 0
brk(0x0808B910)                    = 0
llseek(3, 0, SEEK_CUR)                = 0
llseek(3, 0, SEEK_SET)                = 0
fstat64(3, 0x080475A0)                = 0
brk(0x0808B910)                    = 0
brk(0x08093910)                    = 0
fstat64(3, 0x080474B0)                = 0
ioctl(3, TCGETA, 0x08047550)            Err#25 ENOTTY
read(3, " R u n e M a g 1 U T F -".., 30208)    = 30188
brk(0x08093910)                    = 0
brk(0x0809D910)                    = 0
llseek(3, 0, SEEK_CUR)                = 30188
close(3)                    = 0
open("/usr/lib/locale//en_US.UTF-8/LC_NUMERIC/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x08047590)                = 0
fstat(3, 0x08047B60)                = 0
read(3, " .\n ,\n 3\n", 6)            = 6
close(3)                    = 0
open("/usr/lib/locale//en_US.UTF-8/LC_TIME/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475A0)                = 0
fstat(3, 0x08047B70)                = 0
read(3, " J a n\n F e b\n M a r\n".., 308)    = 308
close(3)                    = 0
open("/usr/lib/locale//en_US.UTF-8/LC_COLLATE/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475C0)                = 0
fstat(3, 0x08047BA0)                = 0
mmap(0x00000000, 75408, PROT_READ, MAP_PRIVATE, 3, 0) = 0xFEDDC000
close(3)                    = 0
open("/usr/lib/locale//en_US.UTF-8/LC_MONETARY/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x08047590)                = 0
fstat(3, 0x08047B60)                = 0
read(3, " U S D  \n $\n .\n ,\n 3".., 44)    = 44
close(3)                    = 0
open("/usr/lib/locale//en_US.UTF-8/LC_MESSAGES/LCL_DATA", O_RDONLY) = 3
fstat64(3, 0x080475A0)                = 0
fstat(3, 0x08047B70)                = 0
read(3, " ^ ( ( [ y Y ] ( [ e E ]".., 45)    = 45
close(3)                    = 0
ioctl(1, TCGETA, 0x08047C50)            = 0
ioctl(1, TIOCGWINSZ, 0x08047CB8)        = 0
open(".", O_RDONLY|O_NDELAY|O_LARGEFILE)    = 3
fstat64(3, 0x08047940)                = 0
fcntl(3, F_SETFD, 0x00000001)            = 0
fstat64(3, 0x08047A40)                = 0
getdents64(3, 0xFEDF4000, 8192)            = 528
getdents64(3, 0xFEDF4000, 8192)            = 0
close(3)                    = 0
close(1)                    = 0
close(2)                    = 0
_exit(0)
truss ls  0.00s user 0.00s system 0% cpu 26.226 total
-(~)-[✓]-{ vmstat 5 5                                                                                                 }-(sjorge@gravity)-
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr s0 s3 s9 s1   in   sy   cs us sy id
 1 0 0 34207128 21780212 7 34 0 0  0  0 26  3 -0  1  1 1436  868 1100  0  0 100
 0 0 0 30331980 17890896 7 14 0 0  0  0  0  0  0  4  4 1269  375  949  0  0 100
 1 0 0 30331868 17890816 0 0 0  0  0  0  0  0  0  0  0 1087  368  814  0  0 100
 1 0 0 30331868 17890816 0 0 0  0  0  0  0  0  0  0  0 1092  405  824  0  0 100
 0 0 0 30331868 17890816 0 0 0  0  0  0  0  0  0  0  0 1141  354  808  0  0 100
-(~)-[✓]-{ mpstat 5 5                                                                                                 }-(sjorge@gravity)-
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    2   0    3   455  219   85    0    2    4    0    61    0   0   0  99
  1    5   0    8   113   33  134    0    2    4    0   103    0   0   0  99
  2    5   0    5    96   25  110    0    2    3    0    95    0   0   0  99
  3    2   0    3    79   23   85    0    1    3    0    47    0   0   0 100
  4    2   0    2    61   14   64    0    1    3    0    67    0   0   0 100
  5    3   0    5    86   21  100    0    2    3    0    83    0   0   0 100
  6    4   0    5    96   25  109    0    2    3    0    82    0   0   0 100
  7    1   0    2   101   53   67    0    1    5    0    49    0   0   0 100
  8    1   0    2    60   15   61    0    1    2    0    39    0   0   0 100
  9    4   0    5   100   31  109    0    2    4    0   109    0   0   0 100
 10    5   0    6   104   35  109    0    2    4    0    90    0   0   0 100
 11    1   0    2    83   36   65    0    1    3    0    37    0   0   0 100
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0    0   333  110   56    0    2    1    0    17    0   0   0 100
  1    0   0    7    85   20   96    0    3    4    0    40    0   0   0 100
  2    0   0    0    85   30   90    0    3    6    0    46    0   1   0  99
  3    0   0    9    53   29   34    0    2    3    0    17    0   0   0 100
  4    0   0    0    86   32   94    0    1    2    0    30    0   0   0 100
  5    0   0    9   127   55  136    0    2    5    0    16    0   0   0 100
  6    1   0    0    67    5   59    0    3    2    0    71    0   0   0 100
  7    0   0    0    89   27   77    0    1    2    0    89    0   0   0 100
  8    0   0    2    96   27   83    0    1    2    0    27    0   0   0 100
  9    0   0    9    60   26   89    0    1    4    0    90    0   0   0 100
 10    0   0    2    65   16   60    0    1    4    0    41    0   0   0 100
 11    0   0    0    51   11   38    0    1    1    0    10    0   0   0 100
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0    0   315  105   24    0    0    0    0     6    0   0   0 100
  1    0   0    0    49    9   41    0    1    0    0    23    0   0   0 100
  2    0   0    0    53   12   47    0    1    0    0    16    0   1   0  99
  3    0   0    0    42    9   40    0    1    0    0     8    0   0   0 100
  4    0   0    0    63   19   60    0    2    0    0     5    0   0   0 100
  5    0   0    4   125   54  121    0    1    0    0     9    0   0   0 100
  6    1   0    4    58   12   59    0    1    0    0    89    0   0   0 100
  7    0   0    0    35    3   25    0    1    0    0    11    0   0   0 100
  8    0   0    0   130   48  119    0    0    0    0    47    0   0   0 100
  9    0   0    4    96   42  103    0    1    0    0    60    0   0   0 100
 10    0   0    4   129   49  135    0    1    1    0    83    0   0   0 100
 11    0   0    0    32    6   19    0    0    0    0     3    0   0   0 100
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0    4   331  105   72    0    2    0    0    20    0   0   0 100
  1  141   0    9    74   19   77    0    3    1    0   110    0   0   0 100
  2    0   0   12    94   31   96    0    2    0    0    38    0   0   0  99
  3    0   0    0    66   16   58    0    2    1    0    22    0   0   0 100
  4    0   0   11    51    7   43    0    1    1    0    11    0   1   0  99
  5   69   0   11    93   35   81    0    1    1    0    35    0   0   0 100
  6   80   0    4    48    4   32    0    1    1    0   103    0   0   0 100
  7    0   0    0    28    2   14    0    0    0    0     4    0   0   0 100
  8    0   0   18   124   50  111    0    0    0    0    49    0   0   0 100
  9    0   0    9    91   35   78    0    1    0    0    27    0   0   0 100
 10  232   0   18   126   42  128    0    1    1    0   798    1   0   0  99
 11    0   0    0    39    5   20    0    1    0    0     4    0   0   0 100
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0    0   331  105   45    0    1    2    0    12    0   0   0 100
  1    0   0    7    92   26  110    0    1    3    0    66    0   0   0 100
  2    0   0    4    38    7   38    0    2    3    0     9    0   1   0  99
  3    0   0   13    76   34   58    0    1    1    0    34    0   0   0 100
  4    0   0    0    47    3   37    0    1    1    0     5    0   0   0 100
  5    0   0    4    63   17   62    0    2    2    0    15    0   0   0 100
  6    0   0    0    72   16   69    0    3    2    0    89    0   0   0 100
  7    0   0   11    46   10   42    0    2    3    0    16    0   0   0 100
  8    0   0   13   152   59  141    0    2    1    0    52    0   0   0 100
  9    0   0   18   208   95  200    0    1    1    0    47    0   0   0 100
 10    0   0    4    81   15   65    0    1    3    0    10    0   0   0 100
 11    0   0    0    28    3   15    0    1    1    0     4    0   0   0 100
-(~)-[✗]-{ intrstat                                                                                                     }-(root@gravity)-

      device |      cpu0 %tim      cpu1 %tim      cpu2 %tim      cpu3 %tim      cpu4 %tim      cpu5 %tim      cpu6 %tim      cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0       147  0.2
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0       147  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#3 |         1  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0       147  0.0
      ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0       147  0.2

      device |      cpu8 %tim      cpu9 %tim     cpu10 %tim     cpu11 %tim
-------------+------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         1  0.0
    e1000g#1 |         0  0.0        15  0.0         0  0.0         0  0.0
    e1000g#3 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0        12  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0

      device |      cpu0 %tim      cpu1 %tim      cpu2 %tim      cpu3 %tim      cpu4 %tim      cpu5 %tim      cpu6 %tim      cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0         2  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         2  0.0
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         2  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#3 |         2  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         2  0.0
      ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         2  0.0

      device |      cpu8 %tim      cpu9 %tim     cpu10 %tim     cpu11 %tim
-------------+------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         2  0.0
    e1000g#1 |         0  0.0        22  0.0         0  0.0         0  0.0
    e1000g#3 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0        16  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#1 |         1  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0

      device |      cpu0 %tim      cpu1 %tim      cpu2 %tim      cpu3 %tim      cpu4 %tim      cpu5 %tim      cpu6 %tim      cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#3 |         1  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0
      ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0

      device |      cpu8 %tim      cpu9 %tim     cpu10 %tim     cpu11 %tim
-------------+------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         1  0.0
    e1000g#1 |         0  0.0        18  0.0         0  0.0         0  0.0
    e1000g#3 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0        16  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0

      device |      cpu0 %tim      cpu1 %tim      cpu2 %tim      cpu3 %tim      cpu4 %tim      cpu5 %tim      cpu6 %tim      cpu7 %tim
-------------+------------------------------------------------------------------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0        88  0.1         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#3 |         1  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0
      ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         0  0.0         1  0.0

      device |      cpu8 %tim      cpu9 %tim     cpu10 %tim     cpu11 %tim
-------------+------------------------------------------------------------
      ahci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#0 |         0  0.0         0  0.0         0  0.0         0  0.0
    arcsas#2 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#0 |         0  0.0         0  0.0         0  0.0         1  0.0
    e1000g#1 |         0  0.0        21  0.0         0  0.0         0  0.0
    e1000g#3 |         0  0.0         0  0.0         0  0.0         0  0.0
    e1000g#4 |         0  0.0         0  0.0        15  0.0         0  0.0
      ehci#0 |         0  0.0         0  0.0         0  0.0         0  0.0
      ehci#1 |         0  0.0         0  0.0         0  0.0         0  0.0
    nvidia#1 |         0  0.0         0  0.0         0  0.0         0  0.0
-(DTT)-[✗]-{ dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'                                                 }-(root@gravity)-
dtrace: description 'syscall:::entry ' matched 231 probes
^X^C

  getloadavg                                                        1
  kill                                                              1
  lwp_detach                                                        1
  lwp_mutex_timedlock                                               1
  lwp_mutex_unlock                                                  1
  mmap                                                              1
  setsockopt                                                        1
  shutdown                                                          1
  sigpending                                                        1
  sendto                                                            2
  umount2                                                           2
  statvfs64                                                         3
  sysconfig                                                         3
  alarm                                                             4
  fstat                                                             4
  getsockname                                                       4
  pause                                                             4
  recvfrom                                                          4
  sigaction                                                         4
  openat                                                            6
  putmsg                                                            6
  rename                                                            6
  brk                                                               8
  accept                                                           11
  getdents64                                                       12
  lwp_continue                                                     13
  lwp_create                                                       13
  lwp_exit                                                         13
  uucopy                                                           13
  pread                                                            14
  schedctl                                                         14
  clock_gettime                                                    16
  lwp_kill                                                         17
  getpeername                                                      18
  unlink                                                           18
  send                                                             23
  c2audit                                                          25
  portfs                                                           25
  getuid                                                           36
  writev                                                           37
  getsockopt                                                       40
  connect                                                          42
  so_socket                                                        49
  open                                                             62
  stat                                                             71
  fdsync                                                           72
  open64                                                           72
  getmsg                                                           75
  stat64                                                           76
  sendmsg                                                          92
  recvmsg                                                          95
  pset                                                            141
  pwrite64                                                        180
  close                                                           225
  waitsys                                                         237
  sigsuspend                                                      238
  p_online                                                        256
  setcontext                                                      264
  nanosleep                                                       276
  pread64                                                         503
  access                                                          960
  lwp_sigmask                                                     960
  read                                                           1103
  write                                                          1177
  llseek                                                         1210
  fstat64                                                        1546
  doorfs                                                         1614
  getpid                                                         1667
  fcntl                                                          2304
  gtime                                                          2422
  recv                                                           5302
  lwp_park                                                      23651
  pollsys                                                       44133
  ioctl                                                         64358
(DTT)-[✓]-{ ./hotkernel                                                                                                }-(root@gravity)-
Sampling... Hit Ctrl-C to end.
^C
FUNCTION                                                COUNT   PCNT
unix`page_unlock                                            1   0.0%
genunix`thread_free_prevent                                 1   0.0%
aggr`aggr_find_tx_ring                                      1   0.0%
unix`gdt_ucode_model                                        1   0.0%
arcsas`arcsas_build_os_ccb                                  1   0.0%
genunix`new_mstate                                          1   0.0%
genunix`clock_tick                                          1   0.0%
SDC`sysdc_wakeup                                            1   0.0%
genunix`callout_list_get                                    1   0.0%
unix`xsave_ctxt                                             1   0.0%
unix`reset_turbo_info                                       1   0.0%
zfs`zio_vdev_io_done                                        1   0.0%
genunix`taskq_ent_alloc                                     1   0.0%
sha1`sha1_block_data_order                                  1   0.0%
sd`sdstrategy                                               1   0.0%
genunix`kmem_cache_alloc                                    1   0.0%
unix`set_freemem                                            1   0.0%
unix`cpupm_wakeup_cstate_data                               1   0.0%
genunix`pcache_poll                                         1   0.0%
genunix`callout_list_expire                                 1   0.0%
genunix`lwp_hash_lookup_and_lock                            1   0.0%
unix`rw_exit                                                1   0.0%
stmf`stmf_svc_timeout                                       1   0.0%
arcsas`arcsas_device_get_sent_ccb_from_list                 1   0.0%
zfs`vdev_queue_io_remove                                    1   0.0%
rootnex`rootnex_coredma_allochdl                            1   0.0%
genunix`copyin_args32                                       1   0.0%
genunix`callout_downheap                                    1   0.0%
zfs`vdev_queue_offset_compare                               1   0.0%
genunix`setrun_locked                                       1   0.0%
unix`strcpy                                                 1   0.0%
unix`cpu_resched                                            1   0.0%
unix`_sys_sysenter_post_swapgs                              1   0.0%
unix`bcopy                                                  1   0.0%
genunix`disp_lock_exit_high                                 1   0.0%
genunix`avl_walk                                            1   0.0%
unix`av_dispatch_softvect                                   1   0.0%
zfs`dmu_object_size_from_db                                 1   0.0%
TS`ts_sleep                                                 1   0.0%
genunix`mexchange                                           1   0.0%
arcsas`arcsas_ccb_done                                      1   0.0%
genunix`callout_expire                                      1   0.0%
ip`squeue_worker                                            1   0.0%
arcsas`arcsas_check_irq_state                               1   0.0%
zfs`buf_hash_find                                           1   0.0%
unix`sep_restore                                            1   0.0%
unix`prunstop                                               1   0.0%
sd`sdintr                                                   1   0.0%
unix`xc_call                                                1   0.0%
zfs`arc_buf_remove_ref                                      1   0.0%
genunix`syscall_exit                                        1   0.0%
zfs`space_map_seg_compare                                   1   0.0%
genunix`cyclic_reprogram_cyclic                             1   0.0%
genunix`taskq_thread                                        1   0.0%
pcplusmp`apic_unset_idlecpu                                 1   0.0%
genunix`cv_waituntil_sig                                    1   0.0%
genunix`schedctl_set_cidpri                                 1   0.0%
unix`x86_hm_exit                                            1   0.0%
genunix`cv_wait                                             1   0.0%
genunix`fd_find                                             1   0.0%
unix`cpupm_utilization_event                                1   0.0%
zfs`dbuf_hash                                               1   0.0%
pcplusmp`apic_send_ipi                                      1   0.0%
unix`cpupm_state_change                                     1   0.0%
genunix`schedctl_restore                                    1   0.0%
genunix`ddi_dma_sync                                        1   0.0%
unix`atomic_add_64_nv                                       1   0.0%
genunix`list_remove                                         1   0.0%
zfs`dnode_block_freed                                       1   0.0%
genunix`avl_numnodes                                        1   0.0%
unix`hat_kpm_pfn2va                                         1   0.0%
genunix`sleepq_insert                                       1   0.0%
genunix`expire_tick                                         1   0.0%
unix`idle_exit                                              1   0.0%
unix`av_clear_softint_pending                               1   0.0%
genunix`cv_wait_sig_swap_core                               1   0.0%
unix`rdmsr                                                  1   0.0%
genunix`turnstile_block                                     1   0.0%
genunix`cv_unsleep                                          1   0.0%
genunix`sys_nullsys                                         1   0.0%
unix`clock_tick_process                                     1   0.0%
arcsas`arcsas_set_lba_and_sector_count                      1   0.0%
unix`atomic_add_64                                          1   0.0%
unix`atomic_add_32_nv                                       1   0.0%
unix`rw_enter                                               2   0.0%
genunix`poll_common                                         2   0.0%
unix`gdt_update_usegd                                       2   0.0%
genunix`cpu_update_pct                                      2   0.0%
unix`hat_switch                                             2   0.0%
unix`lock_try                                               2   0.0%
unix`wrmsr                                                  2   0.0%
genunix`cyclic_softint                                      2   0.0%
unix`sep_save                                               2   0.0%
unix`atomic_cas_32                                          2   0.0%
unix`setbackdq                                              2   0.0%
genunix`untimeout_default                                   2   0.0%
genunix`thread_lock                                         2   0.0%
unix`cpu_acpi_idle                                          2   0.0%
unix`bitset_atomic_add                                      2   0.0%
unix`cpupm_next_cstate                                      2   0.0%
unix`_resume_from_idle                                      2   0.0%
genunix`ddi_get_lbolt                                       2   0.0%
unix`clock_tick_execute_common                              2   0.0%
unix`lwp_segregs_restore32                                  2   0.0%
genunix`cv_timedwait_sig_hires                              2   0.0%
unix`lock_set_spl                                           2   0.0%
genunix`taskq_dispatch                                      2   0.0%
mac`mac_rx_deliver                                          2   0.0%
genunix`callout_heap_delete                                 3   0.0%
unix`mutex_delay_default                                    3   0.0%
unix`mul32                                                  3   0.0%
zfs`l2arc_write_buffers                                     3   0.0%
unix`0xfffffffffb85                                         3   0.0%
unix`disp_anywork                                           3   0.0%
unix`mutex_tryenter                                         3   0.0%
genunix`clock                                               3   0.0%
unix`default_lock_delay                                     3   0.0%
unix`tsc_scalehrtime                                        3   0.0%
unix`cmt_ev_thread_swtch                                    3   0.0%
unix`atomic_and_64                                          3   0.0%
unix`splr                                                   3   0.0%
unix`disp_ratify                                            3   0.0%
genunix`syscall_entry                                       3   0.0%
genunix`timeout_generic                                     3   0.0%
dtrace`dtrace_dynvar_clean                                  3   0.0%
rootnex`rootnex_teardown_copybuf                            3   0.0%
unix`bzero                                                  3   0.0%
genunix`post_syscall                                        4   0.0%
unix`cmt_balance                                            4   0.0%
unix`mutex_owned                                            4   0.0%
unix`mutex_exit                                             4   0.0%
unix`page_pptonum                                           4   0.0%
unix`lock_set                                               4   0.0%
unix`idle                                                   4   0.0%
unix`bitset_in_set                                          4   0.0%
sha2`SHA256TransformBlocks                                  5   0.0%
unix`swtch                                                  5   0.0%
unix`resume                                                 5   0.0%
unix`dispatch_hilevel                                       5   0.0%
unix`ddi_get32                                              5   0.0%
unix`membar_enter                                           5   0.0%
unix`prefetch_page_r                                        6   0.0%
genunix`syscall_mstate                                      6   0.0%
genunix`restorectx                                          6   0.0%
unix`av_check_softint_pending                               7   0.0%
unix`atomic_add_32                                          8   0.0%
zfs`lzjb_compress                                           9   0.0%
unix`atomic_or_64                                           9   0.0%
unix`tsc_gethrtime_delta                                   10   0.0%
zfs`l2arc_write_eligible                                   10   0.0%
genunix`lbolt_event_driven                                 11   0.0%
unix`mutex_enter                                           13   0.0%
genunix`list_next                                          14   0.0%
unix`disp_getwork                                          18   0.0%
unix`tsc_gethrtimeunscaled_delta                           19   0.0%
unix`dispatch_softint                                      26   0.0%
zfs`buf_hash                                               27   0.0%
unix`do_splx                                               47   0.0%
unix`page_nextn                                           103   0.0%
genunix`fsflush_do_pages                                  108   0.0%
unix`i86_mwait                                            601   0.1%
unix`acpi_cpu_cstate                                   405605  99.7%
-(~adm)-[✓]-{ tail -n200 messages                                                                                       }-(root@gravity)-
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hubd1 is /pci@0,0/pci8086,4953@1d/hub@1
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1d/hub@1 (hubd1) online
Aug 15 20:09:47 gravity.acheron.be drmach_acpi: [ID 779328 kern.warning] WARNING: drmach_init: number of logical CPUs (12) in physical processor is not power of 2.
Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 device (usb4f3,103) operating at low speed (USB 1.x) on USB 2.0 external hub: device@5, usb_mid1 at bus address 3
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] usb_mid1 is /pci@0,0/pci8086,4953@1a/hub@1/device@5
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5 (usb_mid1) online
Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 interface (usbif4f3,103.config1.0) operating at low speed (USB 1.x) on USB 2.0 external hub: keyboard@0, hid4 at bus address 3
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hid4 is /pci@0,0/pci8086,4953@1a/hub@1/device@5/keyboard@0
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5/keyboard@0 (hid4) online
Aug 15 20:09:47 gravity.acheron.be usba: [ID 912658 kern.info] USB 1.10 interface (usbif4f3,103.config1.1) operating at low speed (USB 1.x) on USB 2.0 external hub: input@1, hid5 at bus address 3
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 936769 kern.info] hid5 is /pci@0,0/pci8086,4953@1a/hub@1/device@5/input@1
Aug 15 20:09:47 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,4953@1a/hub@1/device@5/input@1 (hid5) online
Aug 15 20:09:48 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: audio0
Aug 15 20:09:48 gravity.acheron.be genunix: [ID 936769 kern.info] audio0 is /pseudo/audio@0
Aug 15 20:09:51 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10d3 (e1000g) instance 1 irq 0x34 vector 0x60 ioapic 0xff intin 0xff is bound to cpu 9
Aug 15 20:09:51 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g1 registered
Aug 15 20:09:51 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,107d (e1000g) instance 4 irq 0x35 vector 0x61 ioapic 0xff intin 0xff is bound to cpu 10
Aug 15 20:09:51 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g4 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: aggr1000 registered
Aug 15 20:09:52 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pci8086,1502 (e1000g) instance 0 irq 0x36 vector 0x62 ioapic 0xff intin 0xff is bound to cpu 11
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g0 registered
Aug 15 20:09:52 gravity.acheron.be pcplusmp: [ID 805372 kern.info] pcplusmp: pciex8086,10d3 (e1000g) instance 3 irq 0x37 vector 0x63 ioapic 0xff intin 0xff is bound to cpu 0
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: e1000g3 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: aggr1001 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1002 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1003 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1002 link up, 0 Mbps, unknown duplex
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1004 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1005 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1006 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1007 registered
Aug 15 20:09:52 gravity.acheron.be mac: [ID 469746 kern.info] NOTICE: vnic1008 registered
Aug 15 20:09:53 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Aug 15 20:09:53 gravity.acheron.be genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Aug 15 20:09:54 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: zfs0
Aug 15 20:09:54 gravity.acheron.be genunix: [ID 936769 kern.info] zfs0 is /pseudo/zfs@0
Aug 15 20:09:54 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g4 link up, 1000 Mbps, full duplex
Aug 15 20:09:54 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: aggr1000 link up, 1000 Mbps, full duplex
Aug 15 20:09:54 gravity.acheron.be drmach_acpi: [ID 779328 kern.warning] WARNING: drmach_init: number of logical CPUs (12) in physical processor is not power of 2.
Aug 15 20:09:55 gravity.acheron.be arcsas: [ID 768306 kern.notice] 
Aug 15 20:09:55 gravity.acheron.be arcsas2:Areca SAS 6G Host Adapter  
Aug 15 20:09:55 gravity.acheron.be arcsas: [ID 768306 kern.notice] arcsas2:ARC13x0 driver v1.00.00.02 2012-03-23 
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g1 link up, 1000 Mbps, full duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g3 link up, 1000 Mbps, full duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: aggr1001 link up, 1000 Mbps, full duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1007 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1008 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1005 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1006 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1004 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:55 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: vnic1003 link up, 1000 Mbps, unknown duplex
Aug 15 20:09:56 gravity.acheron.be mac: [ID 435574 kern.info] NOTICE: e1000g0 link up, 1000 Mbps, full duplex
Aug 15 20:10:25 gravity.acheron.be unix: [ID 954099 kern.info] NOTICE: IRQ16 is being shared by drivers with different interrupt levels.
Aug 15 20:10:25 gravity.acheron.be This may result in reduced system performance.
Aug 15 20:10:27 gravity.acheron.be pcieb: [ID 586369 kern.info] PCIE-device: pci17d3,1320@0, arcsas2
Aug 15 20:10:27 gravity.acheron.be npe: [ID 236367 kern.info] PCI Express-device: pci17d3,1320@0, arcsas2
Aug 15 20:10:27 gravity.acheron.be genunix: [ID 936769 kern.info] arcsas2 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd20 at arcsas2: target 1 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd20 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@1,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@1,0 (sd20) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 1 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd22 at arcsas2: target 3 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd22 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@3,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@3,0 (sd22) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 3 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd19 at arcsas2: target 0 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd19 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@0,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@0,0 (sd19) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 0 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd21 at arcsas2: target 2 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd21 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@2,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@2,0 (sd21) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 2 LUN 0 onlined
Aug 15 20:10:28 gravity.acheron.be scsi: [ID 583861 kern.info] sd23 at arcsas2: target 4 lun 0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 936769 kern.info] sd23 is /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@4,0
Aug 15 20:10:28 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c04@2/pci17d3,1320@0/disk@4,0 (sd23) online
Aug 15 20:10:28 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas2: Target 4 LUN 0 onlined
Aug 15 20:10:29 gravity.acheron.be arcsas: [ID 768306 kern.notice] 
Aug 15 20:10:29 gravity.acheron.be arcsas0:Areca SAS 6G Host Adapter  
Aug 15 20:10:29 gravity.acheron.be arcsas: [ID 768306 kern.notice] arcsas0:ARC13x0 driver v1.00.00.02 2012-03-23 
Aug 15 20:10:54 gravity.acheron.be unix: [ID 954099 kern.info] NOTICE: IRQ16 is being shared by drivers with different interrupt levels.
Aug 15 20:10:54 gravity.acheron.be This may result in reduced system performance.
Aug 15 20:10:55 gravity.acheron.be svc.startd[10]: [ID 122153 daemon.warning] svc:/system/boot-archive:default: Method or service exit timed out.  Killing contract 16.
Aug 15 20:10:56 gravity.acheron.be svc.startd[10]: [ID 636263 daemon.warning] svc:/system/boot-archive:default: Method "/lib/svc/method/boot-archive" failed due to signal KILL.
Aug 15 20:10:56 gravity.acheron.be pcieb: [ID 586369 kern.info] PCIE-device: pci17d3,1320@0, arcsas0
Aug 15 20:10:56 gravity.acheron.be npe: [ID 236367 kern.info] PCI Express-device: pci17d3,1320@0, arcsas0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] arcsas0 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0
Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd17 at arcsas0: target 5 lun 0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd17 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@5,0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@5,0 (sd17) online
Aug 15 20:10:56 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 5 LUN 0 onlined
Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd12 at arcsas0: target 0 lun 0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd12 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@0,0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@0,0 (sd12) online
Aug 15 20:10:56 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 0 LUN 0 onlined
Aug 15 20:10:56 gravity.acheron.be scsi: [ID 583861 kern.info] sd14 at arcsas0: target 2 lun 0
Aug 15 20:10:56 gravity.acheron.be genunix: [ID 936769 kern.info] sd14 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@2,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@2,0 (sd14) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 2 LUN 0 onlined
Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd15 at arcsas0: target 3 lun 0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd15 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@3,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@3,0 (sd15) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 3 LUN 0 onlined
Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd13 at arcsas0: target 1 lun 0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd13 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@1,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@1,0 (sd13) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 1 LUN 0 onlined
Aug 15 20:10:57 gravity.acheron.be scsi: [ID 583861 kern.info] sd16 at arcsas0: target 4 lun 0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 936769 kern.info] sd16 is /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@4,0
Aug 15 20:10:57 gravity.acheron.be genunix: [ID 408114 kern.info] /pci@0,0/pci8086,3c0a@3,2/pci17d3,1320@0/disk@4,0 (sd16) online
Aug 15 20:10:57 gravity.acheron.be arcsas: [ID 642126 kern.notice] NOTICE: arcsas0: Target 4 LUN 0 onlined
Aug 15 20:11:01 gravity.acheron.be genunix: [ID 227219 kern.info] This Solaris instance has UUID 05da9d90-3687-e683-92e2-a79fd35f7b07
Aug 15 20:11:08 gravity.acheron.be genunix: [ID 454863 kern.info] dump on /dev/zvol/dsk/rpool/dump size 16346 MB
Aug 15 20:11:09 gravity.acheron.be rootnex: [ID 349649 kern.info] npe1 at root: space 35 offset 0
Aug 15 20:11:09 gravity.acheron.be genunix: [ID 936769 kern.info] npe1 is /pci@35,0
Aug 15 20:11:09 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pm0
Aug 15 20:11:09 gravity.acheron.be genunix: [ID 936769 kern.info] pm0 is /pseudo/pm@0
Aug 15 20:11:10 gravity.acheron.be rootnex: [ID 349649 kern.info] iscsi0 at root
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] iscsi0 is /iscsi
Aug 15 20:11:10 gravity.acheron.be acpinex: [ID 328922 kern.info] acpinex: sb@0, acpinex1
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] acpinex1 is /fw/sb@0
Aug 15 20:11:10 gravity.acheron.be isa: [ID 202937 kern.info] ISA-device: pit_beep0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] pit_beep0 is /pci@0,0/isa@1f/pit_beep
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: power0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] power0 is /pseudo/power@0
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: srn0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] srn0 is /pseudo/srn@0
Aug 15 20:11:10 gravity.acheron.be /usr/lib/power/powerd: [ID 387247 daemon.error] Able to open /dev/srn
Aug 15 20:11:10 gravity.acheron.be ntpd[300]: [ID 702911 daemon.notice] ntpd 4.2.5p200@1.1948-o Wed Jun  6 13:08:39 BST 2012 (1)
Aug 15 20:11:10 gravity.acheron.be ntpd[332]: [ID 702911 daemon.notice] proto: precision = 0.252 usec
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pseudo1
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] pseudo1 is /pseudo/zconsnex@1
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fct0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] fct0 is /pseudo/fct@0
Aug 15 20:11:10 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: stmf0
Aug 15 20:11:10 gravity.acheron.be genunix: [ID 936769 kern.info] stmf0 is /pseudo/stmf@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: llc10
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] llc10 is /pseudo/llc1@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lofi0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] lofi0 is /pseudo/lofi@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: ramdisk1024
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] ramdisk1024 is /pseudo/ramdisk@1024
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: ucode0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] ucode0 is /pseudo/ucode@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dcpc0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] dcpc0 is /pseudo/dcpc@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dtrace0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] dtrace0 is /pseudo/dtrace@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fasttrap0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fasttrap0 is /pseudo/fasttrap@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fbt0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fbt0 is /pseudo/fbt@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lockstat0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] lockstat0 is /pseudo/lockstat@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: profile0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] profile0 is /pseudo/profile@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: sdt0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] sdt0 is /pseudo/sdt@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: systrace0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] systrace0 is /pseudo/systrace@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fcsm0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fcsm0 is /pseudo/fcsm@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: nvidia255
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] nvidia255 is /pseudo/nvidia@255
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fcp0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fcp0 is /pseudo/fcp@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: pool0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] pool0 is /pseudo/pool@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fssnap0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] fssnap0 is /pseudo/fssnap@0
Aug 15 20:11:11 gravity.acheron.be ipf: [ID 774698 kern.info] IP Filter: v4.1.9, running.
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: nsmb0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] nsmb0 is /pseudo/nsmb@0
Aug 15 20:11:11 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: bpf0
Aug 15 20:11:11 gravity.acheron.be genunix: [ID 936769 kern.info] bpf0 is /pseudo/bpf@0
Aug 15 20:11:12 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: kvm0
Aug 15 20:11:12 gravity.acheron.be genunix: [ID 936769 kern.info] kvm0 is /pseudo/kvm@0
Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@0 (zcons0) online
Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@1 (zcons1) online
Aug 15 20:11:20 gravity.acheron.be genunix: [ID 408114 kern.info] /pseudo/zconsnex@1/zcons@2 (zcons2) online
Aug 15 20:11:24 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
Aug 15 20:11:24 gravity.acheron.be genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
Aug 16 04:11:26 gravity.acheron.be smbsrv: [ID 138215 kern.notice] NOTICE: smbd[GRAVITY\htpc]: download share not found
Aug 16 04:11:26 gravity.acheron.be last message repeated 1 time
Aug 16 10:33:08 gravity.acheron.be mDNSResponder: [ID 702911 daemon.error] ERROR: getOptRdata - unknown opt 4
Aug 16 10:34:02 gravity.acheron.be last message repeated 26 times
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: dcpc0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] dcpc0 is /pseudo/dcpc@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fasttrap0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] fasttrap0 is /pseudo/fasttrap@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: fbt0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] fbt0 is /pseudo/fbt@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: lockstat0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] lockstat0 is /pseudo/lockstat@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: profile0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] profile0 is /pseudo/profile@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: sdt0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] sdt0 is /pseudo/sdt@0
Aug 16 11:06:49 gravity.acheron.be pseudo: [ID 129642 kern.info] pseudo-device: systrace0
Aug 16 11:06:49 gravity.acheron.be genunix: [ID 936769 kern.info] systrace0 is /pseudo/systrace@0

This is all I could grab before the system became so unusable I had to disbale CIFS for a few minutes, back to normal now.
I'd love to grab more specific things if it happens again, but I have little experience debugging stuff on solaris.

#2

Updated by Gordon Ross over 4 years ago

Is this problem still reproducible? If not, let's just close this.

#3

Updated by Jorge Schrauwen over 4 years ago

Hi Gordon,

I no longer have the hardware I had theses issues on.
It went away after I switched to a SuperMicro board + Xeon from an Intel board + i7 combo.

I never did figure out what was causing it aside form disabling CIFS fixed it. (And switched to Samba shortly after)

I guess this can be closed.

Also available in: Atom PDF