Bug #8672
proc_t changes broke genunix dmods and walker
Start date:
2017-09-21
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
Description
Changes made to proc_t in the last couple of years have broken many of
the proc-related dmods and the proc walker when debugging older crash
dumps on newer platforms. Below is the output of running various
proc-related genunix commands on a crash from early 2016 on a platform
from early 2017.
$ thoth debug 8109ce59fb22301cf6d76158ae080f29
thoth: using database at 165.225.169.53:28015 (configured from Manta)
thoth: debugging 8109ce59fb22301cf6d76158ae080f29
* created interactive job -- db116b3b-fb41-e2be-ecd7-99b6edca4d3e
* waiting for session... - established
thoth: dump info is in $THOTH_INFO
mdb: warning: dump is from SunOS 5.11 joyent_20160303T211841Z; dcmds and macros may not match kernel implementation
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba uhci stmf_sbd stmf zfs mm lofs idm mpt_sas sata random cpc logindmux ptm sppp nfs sd ipc ]
> ::status
debugging crash dump /manta/thoth/stor/thoth/8109ce59fb22301cf6d76158ae080f29/vmcore.0 (64-bit) from dal-mappblx001-17
operating system: 5.11 joyent_20160303T211841Z (i86pc)
image uuid: (not set)
panic message: NMI received
dump content: kernel pages only
> !uname -v
joyent_20170202T033902Z
> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 fffffffffbc304c0
R 90 0 0 0 0 0x00020001 ffffff4320466008
R 3 0 0 0 0 0x00020001 ffffff431d409020
R 2 0 0 0 0 0x00020001 ffffff431d40d018
R 1 0 0 0 0 0x4a004000 ffffff431d009010
R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010 /smartlogin-agent
R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught
R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020
R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar
R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught
R 42606 42574 8 8 0 0x4a004000 ffffff433266a000
R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh
R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 ort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u
R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028 -p -c
R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040
R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s
R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug
R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
>
> ::ps -f
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 fffffffffbc304c0
R 90 0 0 0 0 0x00020001 ffffff4320466008
R 3 0 0 0 0 0x00020001 ffffff431d409020
R 2 0 0 0 0 0x00020001 ffffff431d40d018
R 1 0 0 0 0 0x4a004000 ffffff431d009010
R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010
R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 /smartdc/agents/lib/node_modules/cn-age
R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 agent/node/bin/node --abort_on_uncaught
R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020
R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 fig-agent/build/node/bin/node /opt/smar
R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 /smartdc/agents/lib/node_modules/vm-age
R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 agent/node/bin/node --abort_on_uncaught
R 42606 42574 8 8 0 0x4a004000 ffffff433266a000
R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 /smartdc/agents/lib/node_modules/net-ag
R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 -agent/node/bin/node --abort_on_uncaugh
R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 xception /usr/sbin/vmadm lookup -j -o u
R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028
R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040
R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 /smartdc/agents/lib/node_modules/hagfis
R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 fish-watcher/build/node/bin/node /opt/s
R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 /smartdc/agents/lib/node_modules/firewa
R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 ewaller/node/bin/node --abort_on_uncaug
R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 ase/cmd/node cmd/cainstsvc.js /opt/smar
>
> ::pgrep node
S PID PPID PGID SID UID FLAGS ADDR NAME
R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught
R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar
R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught
R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh
R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s
R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug
R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
R 36198 1 8 8 0 0x4a004000 ffffff9c75aff040 -o noorphan /opt/smartdc/agents/lib/node_modules/amon-r
R 36206 36198 8 8 0 0x4a014000 ffffff444d1a7058 agents/lib/node_modules/amon-relay/bin/amon-relay -v -Z
R 36214 36206 8 8 0 0x4a014000 ffffff4331a5e010 /node_modules/amon-relay/build/node/bin/node /opt/smartd
R 35520 1 8 8 0 0x4a004000 ffffff7c56a46050 -o noorphan /opt/smartdc/agents/lib/node_modules/amon-a
R 35521 35520 8 8 0 0x4a014000 ffffff68dea09018 agents/lib/node_modules/amon-agent/bin/amon-agent -v
R 35525 35521 8 8 0 0x4a014000 ffffff8be3b32018 /node_modules/amon-agent/build/node/bin/node /opt/smartd
R 22212 24025 22212 22212 0 0x4a014000 ffffff6d6b74c018 nfig/node_modules/.bin/view-config-service -c /opt/local
R 18127 24045 24045 24045 9416 0x4a014000 ffffff5bc7724040 nfig/node_modules/@walmart/view-config/node_modules/.bin
R 18122 24045 24045 24045 9416 0x4a014000 ffffffb55c9d0048 nfig/node_modules/@walmart/view-config/node_modules/.bin
R 18116 24045 24045 24045 9416 0x4a014000 ffffff5bb5a22040 nfig/node_modules/@walmart/view-config/node_modules/.bin
R 18111 24045 24045 24045 9416 0x4a014000 ffffff5bb85bb038 nfig/node_modules/@walmart/view-config/node_modules/.bin
>
> ::ptree
fffffffffbc304c0
ffffff4320466008
ffffff431d409020
ffffff431d40d018
ffffff431d009010
ffffff4bf98ae010 /smartlogin-agent
ffffff79d0840020 -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught
ffffff7adf68d020
ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar
ffffff79d0922028 -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught
ffffff433266a000
ffffffaafac7d008 -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh
ffffff6d138d3030 ort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u
ffffff5e1c1b2028 -p -c
ffffffefd78a1040
ffffff431dc61038 -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s
ffffff7cc447c040 -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug
ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
ffffff9c75aff040 -o noorphan /opt/smartdc/agents/lib/node_modules/amon-r
>
> ::walk proc | ::print proc_t p_dtrace_count
mdb: proc ffffff5e4417b040 has invalid p_sibling ffffff5e43fd5048; skipping
mdb: proc ffffff6d37b1b030 has invalid p_sibling ffffff5e53720030; skipping
>
> ::cpuinfo
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 fffffffffbc326c0 1f 2 0 60 no no t-2 ffffffaaec8480c0 t/local/penny/node_modules/@walmart/penny/lib/worker.js
1 ffffff42ec7bc040 1f 1 0 169 no no t-8 ffffff01e8443c40
2 fffffffffbc3cfc0 1b 1 0 60 no no t-3 ffffff4445ed2c40 t/local/penny/node_modules/@walmart/penny/lib/worker.js
3 ffffff42ec7a5500 1f 1 0 169 no no t-6 ffffff01e85a0c40
4 ffffff42ec963540 1f 1 0 169 no no t-3 ffffff01e86e6c40
5 ffffff42ec962040 1f 0 0 161 no no t-7 ffffff01e876fc40
6 ffffff42ec960b00 1f 1 0 169 no no t-3 ffffff01e87efc40
7 ffffff42ec95f500 1f 2 0 169 no no t-3 ffffff01e886fc40
8 ffffff42ec95e000 1f 3 0 169 no no t-3 ffffff01e88fec40
9 ffffff42ec958ac0 1f 1 0 169 no no t-3 ffffff01e899fc40
10 ffffff42ec954b00 1f 11 0 169 no no t-3 ffffff01e8a2ec40
11 ffffff42ec94ea80 1f 3 0 168 no no t-3 ffffff01e8bf7c40
12 ffffff42eca63540 1f 14 0 168 no no t-3 ffffff01e8cd9c40
13 ffffff42eca62040 1f 7 0 168 no no t-3 ffffff01e8d50c40
14 ffffff42eca60080 1f 4 0 169 no no t-6 ffffff01e8df1c40
15 ffffff42eca5a000 1f 2 0 169 no no t-3 ffffff01e8fc7c40
16 ffffff42ecb2c080 1f 3 0 161 no no t-8 ffffff01e92d4c40
17 ffffff42ecb2aa80 1f 2 0 169 no no t-3 ffffff01e934bc40
18 ffffff42ecb23540 1f 1 0 165 no no t-3 ffffff01e93d4c40
19 ffffff42ecb22040 1f 3 0 165 no no t-3 ffffff01e946fc40
20 ffffff42ecb1eb00 1f 0 0 165 no no t-3 ffffff01e94f8c40
21 ffffff430ce71500 1f 0 0 165 no no t-0 ffffff01e9566c40
22 ffffff430ce69540 1f 0 0 165 no no t-7 ffffff01e95fec40
>
> ffffff431e166020::pfiles
FD TYPE VNODE INFO
0 CHR ffffff431d5a6b80 /devices/pseudo/mm@0:null
1 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
2 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
3 PORT ffffff7c4d970780 [event port (port=ffffff96eea45a40)]
4 FIFO ffffffffbafb1e40
5 FIFO ffffff444ad1f300
6 REG ffffffaae317c380 /opt/smartdc/agents/log/ca/cainstsvc/amqp-2016-10-19.18:03:53.37508
7 SOCK ffffffaae1584800 socket: AF_INET 192.168.2.167 63975 remote: AF_INET 192.168.2.23 5672
> ffffff431e166020::fd 0
mdb: process ffffff431e166020 only has -1159869698 files open.
>
> ::lminfo
ADDR TP FLAG PID COMM VNODE PATH
ffffff44424f1340 WR 0001 56600 x_agentd ffffffaaf976c000 /zones/16c59be6-8e1
ffffff68de0c6180 WR 0021 51450 ffffffabf9611040 /zones/4a2d66d6-091
ffffffb116282980 WR 0021 24543 ffffffac0477c080 /zones/01fdb772-fea
ffffff4445f2cb00 WR 0021 8084 ffffff444c21c100 /zones/1415a57e-e3b
ffffff4e4cc6d240 WR 0021 92154 ffffffaad99a4100 /zones/d50bbf1d-6d3
ffffffa4b3e57580 WR 0021 23085 ffffffac00f56140 /zones/68fadf2d-fb8
ffffff444a863840 WR 0001 8046 x_agentd ffffff4444491180 /zones/1415a57e-e3b
ffffff4449bc6440 WR 0021 9025 ffffff444c93e280 /zones/c389d77f-ef7
ffffff444389cc40 WR 0021 7769 ffffff4441105300 /zones/16c59be6-8e1
ffffff444930f080 WR 0021 8903 ffffff4449e37340 /zones/2270bbde-8ae
ffffff6d35f7fd40 WR 0001 25464 x_agentd ffffff4461190340 /zones/68fadf2d-fb8
ffffff4333031a80 WR 0001 3059 sendmail -Ac -q15m ffffff4442c61380 /var/spool/clientmq
ffffff443f580540 WR 0021 91301 ffffffaad9e01380 /zones/ee2cddf7-291
ffffff4443547980 WR 0001 3116 sendmail -bl -q15m ffffff444420f400 /var/run/sendmail.p
ffffff443f315340 WR 0021 7881 ffffff4444bb9440 /zones/7167d6f1-554
ffffffffbefbc740 WR 0021 61870 ffffffb84b39a400 /zones/8d230657-048
ffffffb11624ad80 WR 0021 154 ffffffaae892a400 /zones/c1ada27d-2d4
ffffff431dc7cd80 WR 0001 218 ffffff43313ae540 /etc/dev/.devfsadm_
ffffff4447676600 WR 0021 8883 ffffff44443c7580 /zones/2270bbde-8ae
ffffff4446604680 WR 0021 9036 ffffff444ba2e580 /zones/c389d77f-ef7
ffffffa4a9f9b280 WR 0001 64199 x_agentd ffffffffb3c60580 /zones/f65d52b1-137
ffffff44462e5440 WR 0021 92150 ffffff4f1a42f680 /zones/d50bbf1d-6d3
ffffff4449bbe400 WR 0021 7782 ffffff4445bbe700 /zones/16c59be6-8e1
>
> ::walk task_cache ! head -1
0xffffff4480d113c8
> 0xffffff4480d113c8::walk task
0xffffff6d6b74c018
mdb: failed to read proc at 0: no mapping for address
>
> ::walk as_cache ! head -1
0xffffffaad9b732e0
> 0xffffffaad9b732e0::as2proc
mdb: proc ffffff5e4417b040 has invalid p_sibling ffffff5e43fd5048; skipping
mdb: proc ffffff6d37b1b030 has invalid p_sibling ffffff5e53720030; skipping
mdb: proc ffffff5e23aeb010 has invalid p_sibling ffffff5e1be54048; skipping
--
The solution here is to leverage CTF data to get at this.
--
Here is my test output with a patched genunix:
$ thoth debug 8109ce59fb22301cf6d76158ae080f29
thoth: using database at 165.225.169.53:28015 (configured from Manta)
thoth: debugging 8109ce59fb22301cf6d76158ae080f29
* created interactive job -- 501eba91-06c6-eeec-d527-971664d9103f
* waiting for session... \ established
thoth: dump info is in $THOTH_INFO
mdb: warning: dump is from SunOS 5.11 joyent_20160303T211841Z; dcmds and macros may not match kernel implementation
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba uhci stmf_sbd stmf zfs mm lofs idm mpt_sas sata random cpc logindmux ptm sppp nfs sd ipc ]
> ::status
debugging crash dump /manta/thoth/stor/thoth/8109ce59fb22301cf6d76158ae080f29/vmcore.0 (64-bit) from dal-mappblx001-17
operating system: 5.11 joyent_20160303T211841Z (i86pc)
image uuid: (not set)
panic message: NMI received
dump content: kernel pages only
>
> !uname -v
joyent_20170202T033902Z
>
> !mget -o ~/genunix.so ~~/stor/dmod-genunix28.so
~~/stor/dmod-genunix28.so [==============================================================================>] 100% 924.27KB
> !mount -F lofs ~/genunix.so /usr/lib/mdb/kvm/amd64/genunix.so
> ::unload genunix
> ::load genunix
>
> ::ps
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 fffffffffbc304c0 sched
R 90 0 0 0 0 0x00020001 ffffff4320466008 zpool-zones
R 3 0 0 0 0 0x00020001 ffffff431d409020 fsflush
R 2 0 0 0 0 0x00020001 ffffff431d40d018 pageout
R 1 0 0 0 0 0x4a004000 ffffff431d009010 init
R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010 smartlogin
R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 ctrun
R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 node
R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020 zoneevent
R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 node
R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 ctrun
R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 node
R 42606 42574 8 8 0 0x4a004000 ffffff433266a000 zoneevent
R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 ctrun
R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 node
R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 vmadm
R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028 zoneadm
R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040 zoneevent
R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 ctrun
R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 node
R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 ctrun
R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 node
R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 node
>
> ::ps -f
S PID PPID PGID SID UID FLAGS ADDR NAME
R 0 0 0 0 0 0x00000001 fffffffffbc304c0 sched
R 90 0 0 0 0 0x00020001 ffffff4320466008 zpool-zones
R 3 0 0 0 0 0x00020001 ffffff431d409020 fsflush
R 2 0 0 0 0 0x00020001 ffffff431d40d018 pageout
R 1 0 0 0 0 0x4a004000 ffffff431d009010 /sbin/init
R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010 /opt/smartdc/agents/bin/smartlogin-agent
R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age
R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 /opt/smartdc/agents/lib/node_modules/cn-agent/node/bin/node --abort_on_uncaught
R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020 /usr/vm/sbin/zoneevent
R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 /opt/smartdc/agents/lib/node_modules/config-agent/build/node/bin/node /opt/smar
R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age
R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 /opt/smartdc/agents/lib/node_modules/vm-agent/node/bin/node --abort_on_uncaught
R 42606 42574 8 8 0 0x4a004000 ffffff433266a000 /usr/vm/sbin/zoneevent
R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag
R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 /opt/smartdc/agents/lib/node_modules/net-agent/node/bin/node --abort_on_uncaugh
R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 /usr/node/bin/node --abort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u
R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028 /usr/sbin/zoneadm list -p -c
R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040 /usr/vm/sbin/zoneevent
R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis
R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 /opt/smartdc/agents/lib/node_modules/hagfish-watcher/build/node/bin/node /opt/s
R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/firewa
R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 /opt/smartdc/agents/lib/node_modules/firewaller/node/bin/node --abort_on_uncaug
R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 /opt/smartdc/agents/lib/node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar
>
> ::pgrep node
S PID PPID PGID SID UID FLAGS ADDR NAME
R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 node
R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 node
R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 node
R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 node
R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 node
R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 node
R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 node
R 36214 36206 8 8 0 0x4a014000 ffffff4331a5e010 node
R 35525 35521 8 8 0 0x4a014000 ffffff8be3b32018 node
R 22212 24025 22212 22212 0 0x4a014000 ffffff6d6b74c018 node
R 18127 24045 24045 24045 9416 0x4a014000 ffffff5bc7724040 node
R 18122 24045 24045 24045 9416 0x4a014000 ffffffb55c9d0048 node
R 18116 24045 24045 24045 9416 0x4a014000 ffffff5bb5a22040 node
R 18111 24045 24045 24045 9416 0x4a014000 ffffff5bb85bb038 node
R 18102 24045 24045 24045 9416 0x4a014000 ffffff5bb9a8e000 node
R 18101 24045 24045 24045 9416 0x4a014000 ffffffb588a16020 node
R 18085 24045 24045 24045 9416 0x4a014000 ffffff5b934bf038 node
R 18069 24045 24045 24045 9416 0x4a014000 ffffff5bcd892020 node
R 18040 24045 24045 24045 9416 0x4a014000 ffffff5bdeaee048 node
R 80388 50936 50936 50936 9415 0x4a014000 ffffff4454636040 node
R 60532 80388 50936 50936 9415 0x4a014000 ffffff68dea0e000 node
R 60524 80388 50936 50936 9415 0x4a014000 ffffff433191c040 node
R 60519 80388 50936 50936 9415 0x4a014000 ffffffffa688c038 node
>
> ::ptree
fffffffffbc304c0 sched
ffffff4320466008 zpool-zones
ffffff431d409020 fsflush
ffffff431d40d018 pageout
ffffff431d009010 init
ffffff4bf98ae010 smartlogin
ffffff79d0840020 ctrun
ffffffaafa3f4048 node
ffffff7adf68d020 zoneevent
ffffff68dea0f058 node
ffffff79d0922028 ctrun
ffffff444ab23020 node
ffffff433266a000 zoneevent
ffffffaafac7d008 ctrun
ffffff4444147050 node
ffffff6d138d3030 vmadm
ffffff5e1c1b2028 zoneadm
ffffffefd78a1040 zoneevent
ffffff431dc61038 ctrun
ffffff7cbada3020 node
ffffff7cc447c040 ctrun
ffffff7c5675c008 node
ffffff431e166020 node
ffffff9c75aff040 ctrun
>
> ::walk proc | ::print proc_t p_dtrace_count
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
p_dtrace_count = 0
>
> ::cpuinfo
ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 fffffffffbc326c0 1f 2 0 60 no no t-2 ffffffaaec8480c0 node
1 ffffff42ec7bc040 1f 1 0 169 no no t-8 ffffff01e8443c40 sched
2 fffffffffbc3cfc0 1b 1 0 60 no no t-3 ffffff4445ed2c40 node
3 ffffff42ec7a5500 1f 1 0 169 no no t-6 ffffff01e85a0c40 sched
4 ffffff42ec963540 1f 1 0 169 no no t-3 ffffff01e86e6c40 sched
5 ffffff42ec962040 1f 0 0 161 no no t-7 ffffff01e876fc40 sched
6 ffffff42ec960b00 1f 1 0 169 no no t-3 ffffff01e87efc40 sched
7 ffffff42ec95f500 1f 2 0 169 no no t-3 ffffff01e886fc40 sched
8 ffffff42ec95e000 1f 3 0 169 no no t-3 ffffff01e88fec40 sched
9 ffffff42ec958ac0 1f 1 0 169 no no t-3 ffffff01e899fc40 sched
10 ffffff42ec954b00 1f 11 0 169 no no t-3 ffffff01e8a2ec40 sched
11 ffffff42ec94ea80 1f 3 0 168 no no t-3 ffffff01e8bf7c40 sched
12 ffffff42eca63540 1f 14 0 168 no no t-3 ffffff01e8cd9c40 sched
13 ffffff42eca62040 1f 7 0 168 no no t-3 ffffff01e8d50c40 sched
14 ffffff42eca60080 1f 4 0 169 no no t-6 ffffff01e8df1c40 sched
15 ffffff42eca5a000 1f 2 0 169 no no t-3 ffffff01e8fc7c40 sched
16 ffffff42ecb2c080 1f 3 0 161 no no t-8 ffffff01e92d4c40 sched
17 ffffff42ecb2aa80 1f 2 0 169 no no t-3 ffffff01e934bc40 sched
18 ffffff42ecb23540 1f 1 0 165 no no t-3 ffffff01e93d4c40 sched
19 ffffff42ecb22040 1f 3 0 165 no no t-3 ffffff01e946fc40 sched
20 ffffff42ecb1eb00 1f 0 0 165 no no t-3 ffffff01e94f8c40 sched
21 ffffff430ce71500 1f 0 0 165 no no t-0 ffffff01e9566c40 sched
22 ffffff430ce69540 1f 0 0 165 no no t-7 ffffff01e95fec40 sched
>
> ffffff431e166020::pfiles
FD TYPE VNODE INFO
0 CHR ffffff431d5a6b80 /devices/pseudo/mm@0:null
1 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
2 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log
3 PORT ffffff7c4d970780 [event port (port=ffffff96eea45a40)]
4 FIFO ffffffffbafb1e40
5 FIFO ffffff444ad1f300
6 REG ffffffaae317c380 /opt/smartdc/agents/log/ca/cainstsvc/amqp-2016-10-19.18:03:53.37508
7 SOCK ffffffaae1584800 socket: AF_INET 192.168.2.167 63975 remote: AF_INET 192.168.2.23 5672
>
> ffffff431e166020::fd 0
ffffff4331bcb508
>
> ::lminfo
ADDR TP FLAG PID COMM VNODE PATH
ffffff44424f1340 WR 0001 56600 zabbix_agentd ffffffaaf976c000 /zones/16c59be6-8e1
ffffff68de0c6180 WR 0021 51450 sac ffffffabf9611040 /zones/4a2d66d6-091
ffffffb116282980 WR 0021 24543 sac ffffffac0477c080 /zones/01fdb772-fea
ffffff4445f2cb00 WR 0021 8084 sac ffffff444c21c100 /zones/1415a57e-e3b
ffffff4e4cc6d240 WR 0021 92154 ttymon ffffffaad99a4100 /zones/d50bbf1d-6d3
ffffffa4b3e57580 WR 0021 23085 sac ffffffac00f56140 /zones/68fadf2d-fb8
ffffff444a863840 WR 0001 8046 zabbix_agentd ffffff4444491180 /zones/1415a57e-e3b
ffffff4449bc6440 WR 0021 9025 sac ffffff444c93e280 /zones/c389d77f-ef7
ffffff444389cc40 WR 0021 7769 sac ffffff4441105300 /zones/16c59be6-8e1
ffffff444930f080 WR 0021 8903 ttymon ffffff4449e37340 /zones/2270bbde-8ae
ffffff6d35f7fd40 WR 0001 25464 zabbix_agentd ffffff4461190340 /zones/68fadf2d-fb8
ffffff4333031a80 WR 0001 3059 sendmail ffffff4442c61380 /var/spool/clientmq
ffffff443f580540 WR 0021 91301 sac ffffffaad9e01380 /zones/ee2cddf7-291
ffffff4443547980 WR 0001 3116 sendmail ffffff444420f400 /var/run/sendmail.p
ffffff443f315340 WR 0021 7881 sac ffffff4444bb9440 /zones/7167d6f1-554
ffffffffbefbc740 WR 0021 61870 sac ffffffb84b39a400 /zones/8d230657-048
ffffffb11624ad80 WR 0021 154 sac ffffffaae892a400 /zones/c1ada27d-2d4
ffffff431dc7cd80 WR 0001 218 devfsadm ffffff43313ae540 /etc/dev/.devfsadm_
ffffff4447676600 WR 0021 8883 sac ffffff44443c7580 /zones/2270bbde-8ae
ffffff4446604680 WR 0021 9036 ttymon ffffff444ba2e580 /zones/c389d77f-ef7
ffffffa4a9f9b280 WR 0001 64199 zabbix_agentd ffffffffb3c60580 /zones/f65d52b1-137
ffffff44462e5440 WR 0021 92150 sac ffffff4f1a42f680 /zones/d50bbf1d-6d3
ffffff4449bbe400 WR 0021 7782 ttymon ffffff4445bbe700 /zones/16c59be6-8e1
>
> ::walk task_cache ! head -1
0xffffff4480d113c8
> 0xffffff4480d113c8::walk task
0xffffff6d6b74c018
>
> ::walk as_cache ! head -1
0xffffffaad9b732e0
> 0xffffffaad9b732e0::as2proc
ffffff444b845058
Updated by