Bug #8672
proc_t changes broke genunix dmods and walker
Start date:
2017-09-21
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
Description
Changes made to proc_t in the last couple of years have broken many of
the proc-related dmods and the proc walker when debugging older crash
dumps on newer platforms. Below is the output of running various
proc-related genunix commands on a crash from early 2016 on a platform
from early 2017.
$ thoth debug 8109ce59fb22301cf6d76158ae080f29 thoth: using database at 165.225.169.53:28015 (configured from Manta) thoth: debugging 8109ce59fb22301cf6d76158ae080f29 * created interactive job -- db116b3b-fb41-e2be-ecd7-99b6edca4d3e * waiting for session... - established thoth: dump info is in $THOTH_INFO mdb: warning: dump is from SunOS 5.11 joyent_20160303T211841Z; dcmds and macros may not match kernel implementation Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba uhci stmf_sbd stmf zfs mm lofs idm mpt_sas sata random cpc logindmux ptm sppp nfs sd ipc ] > ::status debugging crash dump /manta/thoth/stor/thoth/8109ce59fb22301cf6d76158ae080f29/vmcore.0 (64-bit) from dal-mappblx001-17 operating system: 5.11 joyent_20160303T211841Z (i86pc) image uuid: (not set) panic message: NMI received dump content: kernel pages only > !uname -v joyent_20170202T033902Z > ::ps S PID PPID PGID SID UID FLAGS ADDR NAME R 0 0 0 0 0 0x00000001 fffffffffbc304c0 R 90 0 0 0 0 0x00020001 ffffff4320466008 R 3 0 0 0 0 0x00020001 ffffff431d409020 R 2 0 0 0 0 0x00020001 ffffff431d40d018 R 1 0 0 0 0 0x4a004000 ffffff431d009010 R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010 /smartlogin-agent R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020 R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught R 42606 42574 8 8 0 0x4a004000 ffffff433266a000 R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 ort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028 -p -c R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040 R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 -o noorphan /opt/smartdc/agents/lib/node_modules/firewa R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar > > ::ps -f S PID PPID PGID SID UID FLAGS ADDR NAME R 0 0 0 0 0 0x00000001 fffffffffbc304c0 R 90 0 0 0 0 0x00020001 ffffff4320466008 R 3 0 0 0 0 0x00020001 ffffff431d409020 R 2 0 0 0 0 0x00020001 ffffff431d40d018 R 1 0 0 0 0 0x4a004000 ffffff431d009010 R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010 R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 /smartdc/agents/lib/node_modules/cn-age R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 agent/node/bin/node --abort_on_uncaught R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020 R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 fig-agent/build/node/bin/node /opt/smar R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 /smartdc/agents/lib/node_modules/vm-age R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 agent/node/bin/node --abort_on_uncaught R 42606 42574 8 8 0 0x4a004000 ffffff433266a000 R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 /smartdc/agents/lib/node_modules/net-ag R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 -agent/node/bin/node --abort_on_uncaugh R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 xception /usr/sbin/vmadm lookup -j -o u R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028 R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040 R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 /smartdc/agents/lib/node_modules/hagfis R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 fish-watcher/build/node/bin/node /opt/s R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 /smartdc/agents/lib/node_modules/firewa R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 ewaller/node/bin/node --abort_on_uncaug R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 ase/cmd/node cmd/cainstsvc.js /opt/smar > > ::pgrep node S PID PPID PGID SID UID FLAGS ADDR NAME R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 -o noorphan /opt/smartdc/agents/lib/node_modules/firewa R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar R 36198 1 8 8 0 0x4a004000 ffffff9c75aff040 -o noorphan /opt/smartdc/agents/lib/node_modules/amon-r R 36206 36198 8 8 0 0x4a014000 ffffff444d1a7058 agents/lib/node_modules/amon-relay/bin/amon-relay -v -Z R 36214 36206 8 8 0 0x4a014000 ffffff4331a5e010 /node_modules/amon-relay/build/node/bin/node /opt/smartd R 35520 1 8 8 0 0x4a004000 ffffff7c56a46050 -o noorphan /opt/smartdc/agents/lib/node_modules/amon-a R 35521 35520 8 8 0 0x4a014000 ffffff68dea09018 agents/lib/node_modules/amon-agent/bin/amon-agent -v R 35525 35521 8 8 0 0x4a014000 ffffff8be3b32018 /node_modules/amon-agent/build/node/bin/node /opt/smartd R 22212 24025 22212 22212 0 0x4a014000 ffffff6d6b74c018 nfig/node_modules/.bin/view-config-service -c /opt/local R 18127 24045 24045 24045 9416 0x4a014000 ffffff5bc7724040 nfig/node_modules/@walmart/view-config/node_modules/.bin R 18122 24045 24045 24045 9416 0x4a014000 ffffffb55c9d0048 nfig/node_modules/@walmart/view-config/node_modules/.bin R 18116 24045 24045 24045 9416 0x4a014000 ffffff5bb5a22040 nfig/node_modules/@walmart/view-config/node_modules/.bin R 18111 24045 24045 24045 9416 0x4a014000 ffffff5bb85bb038 nfig/node_modules/@walmart/view-config/node_modules/.bin > > ::ptree fffffffffbc304c0 ffffff4320466008 ffffff431d409020 ffffff431d40d018 ffffff431d009010 ffffff4bf98ae010 /smartlogin-agent ffffff79d0840020 -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age ffffffaafa3f4048 /node_modules/cn-agent/node/bin/node --abort_on_uncaught ffffff7adf68d020 ffffff68dea0f058 /node_modules/config-agent/build/node/bin/node /opt/smar ffffff79d0922028 -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age ffffff444ab23020 /node_modules/vm-agent/node/bin/node --abort_on_uncaught ffffff433266a000 ffffffaafac7d008 -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag ffffff4444147050 /node_modules/net-agent/node/bin/node --abort_on_uncaugh ffffff6d138d3030 ort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u ffffff5e1c1b2028 -p -c ffffffefd78a1040 ffffff431dc61038 -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis ffffff7cbada3020 /node_modules/hagfish-watcher/build/node/bin/node /opt/s ffffff7cc447c040 -o noorphan /opt/smartdc/agents/lib/node_modules/firewa ffffff7c5675c008 /node_modules/firewaller/node/bin/node --abort_on_uncaug ffffff431e166020 /node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar ffffff9c75aff040 -o noorphan /opt/smartdc/agents/lib/node_modules/amon-r > > ::walk proc | ::print proc_t p_dtrace_count mdb: proc ffffff5e4417b040 has invalid p_sibling ffffff5e43fd5048; skipping mdb: proc ffffff6d37b1b030 has invalid p_sibling ffffff5e53720030; skipping > > ::cpuinfo ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 0 fffffffffbc326c0 1f 2 0 60 no no t-2 ffffffaaec8480c0 t/local/penny/node_modules/@walmart/penny/lib/worker.js 1 ffffff42ec7bc040 1f 1 0 169 no no t-8 ffffff01e8443c40 2 fffffffffbc3cfc0 1b 1 0 60 no no t-3 ffffff4445ed2c40 t/local/penny/node_modules/@walmart/penny/lib/worker.js 3 ffffff42ec7a5500 1f 1 0 169 no no t-6 ffffff01e85a0c40 4 ffffff42ec963540 1f 1 0 169 no no t-3 ffffff01e86e6c40 5 ffffff42ec962040 1f 0 0 161 no no t-7 ffffff01e876fc40 6 ffffff42ec960b00 1f 1 0 169 no no t-3 ffffff01e87efc40 7 ffffff42ec95f500 1f 2 0 169 no no t-3 ffffff01e886fc40 8 ffffff42ec95e000 1f 3 0 169 no no t-3 ffffff01e88fec40 9 ffffff42ec958ac0 1f 1 0 169 no no t-3 ffffff01e899fc40 10 ffffff42ec954b00 1f 11 0 169 no no t-3 ffffff01e8a2ec40 11 ffffff42ec94ea80 1f 3 0 168 no no t-3 ffffff01e8bf7c40 12 ffffff42eca63540 1f 14 0 168 no no t-3 ffffff01e8cd9c40 13 ffffff42eca62040 1f 7 0 168 no no t-3 ffffff01e8d50c40 14 ffffff42eca60080 1f 4 0 169 no no t-6 ffffff01e8df1c40 15 ffffff42eca5a000 1f 2 0 169 no no t-3 ffffff01e8fc7c40 16 ffffff42ecb2c080 1f 3 0 161 no no t-8 ffffff01e92d4c40 17 ffffff42ecb2aa80 1f 2 0 169 no no t-3 ffffff01e934bc40 18 ffffff42ecb23540 1f 1 0 165 no no t-3 ffffff01e93d4c40 19 ffffff42ecb22040 1f 3 0 165 no no t-3 ffffff01e946fc40 20 ffffff42ecb1eb00 1f 0 0 165 no no t-3 ffffff01e94f8c40 21 ffffff430ce71500 1f 0 0 165 no no t-0 ffffff01e9566c40 22 ffffff430ce69540 1f 0 0 165 no no t-7 ffffff01e95fec40 > > ffffff431e166020::pfiles FD TYPE VNODE INFO 0 CHR ffffff431d5a6b80 /devices/pseudo/mm@0:null 1 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log 2 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log 3 PORT ffffff7c4d970780 [event port (port=ffffff96eea45a40)] 4 FIFO ffffffffbafb1e40 5 FIFO ffffff444ad1f300 6 REG ffffffaae317c380 /opt/smartdc/agents/log/ca/cainstsvc/amqp-2016-10-19.18:03:53.37508 7 SOCK ffffffaae1584800 socket: AF_INET 192.168.2.167 63975 remote: AF_INET 192.168.2.23 5672 > ffffff431e166020::fd 0 mdb: process ffffff431e166020 only has -1159869698 files open. > > ::lminfo ADDR TP FLAG PID COMM VNODE PATH ffffff44424f1340 WR 0001 56600 x_agentd ffffffaaf976c000 /zones/16c59be6-8e1 ffffff68de0c6180 WR 0021 51450 ffffffabf9611040 /zones/4a2d66d6-091 ffffffb116282980 WR 0021 24543 ffffffac0477c080 /zones/01fdb772-fea ffffff4445f2cb00 WR 0021 8084 ffffff444c21c100 /zones/1415a57e-e3b ffffff4e4cc6d240 WR 0021 92154 ffffffaad99a4100 /zones/d50bbf1d-6d3 ffffffa4b3e57580 WR 0021 23085 ffffffac00f56140 /zones/68fadf2d-fb8 ffffff444a863840 WR 0001 8046 x_agentd ffffff4444491180 /zones/1415a57e-e3b ffffff4449bc6440 WR 0021 9025 ffffff444c93e280 /zones/c389d77f-ef7 ffffff444389cc40 WR 0021 7769 ffffff4441105300 /zones/16c59be6-8e1 ffffff444930f080 WR 0021 8903 ffffff4449e37340 /zones/2270bbde-8ae ffffff6d35f7fd40 WR 0001 25464 x_agentd ffffff4461190340 /zones/68fadf2d-fb8 ffffff4333031a80 WR 0001 3059 sendmail -Ac -q15m ffffff4442c61380 /var/spool/clientmq ffffff443f580540 WR 0021 91301 ffffffaad9e01380 /zones/ee2cddf7-291 ffffff4443547980 WR 0001 3116 sendmail -bl -q15m ffffff444420f400 /var/run/sendmail.p ffffff443f315340 WR 0021 7881 ffffff4444bb9440 /zones/7167d6f1-554 ffffffffbefbc740 WR 0021 61870 ffffffb84b39a400 /zones/8d230657-048 ffffffb11624ad80 WR 0021 154 ffffffaae892a400 /zones/c1ada27d-2d4 ffffff431dc7cd80 WR 0001 218 ffffff43313ae540 /etc/dev/.devfsadm_ ffffff4447676600 WR 0021 8883 ffffff44443c7580 /zones/2270bbde-8ae ffffff4446604680 WR 0021 9036 ffffff444ba2e580 /zones/c389d77f-ef7 ffffffa4a9f9b280 WR 0001 64199 x_agentd ffffffffb3c60580 /zones/f65d52b1-137 ffffff44462e5440 WR 0021 92150 ffffff4f1a42f680 /zones/d50bbf1d-6d3 ffffff4449bbe400 WR 0021 7782 ffffff4445bbe700 /zones/16c59be6-8e1 > > ::walk task_cache ! head -1 0xffffff4480d113c8 > 0xffffff4480d113c8::walk task 0xffffff6d6b74c018 mdb: failed to read proc at 0: no mapping for address > > ::walk as_cache ! head -1 0xffffffaad9b732e0 > 0xffffffaad9b732e0::as2proc mdb: proc ffffff5e4417b040 has invalid p_sibling ffffff5e43fd5048; skipping mdb: proc ffffff6d37b1b030 has invalid p_sibling ffffff5e53720030; skipping mdb: proc ffffff5e23aeb010 has invalid p_sibling ffffff5e1be54048; skipping
--
The solution here is to leverage CTF data to get at this.
--
Here is my test output with a patched genunix: $ thoth debug 8109ce59fb22301cf6d76158ae080f29 thoth: using database at 165.225.169.53:28015 (configured from Manta) thoth: debugging 8109ce59fb22301cf6d76158ae080f29 * created interactive job -- 501eba91-06c6-eeec-d527-971664d9103f * waiting for session... \ established thoth: dump info is in $THOTH_INFO mdb: warning: dump is from SunOS 5.11 joyent_20160303T211841Z; dcmds and macros may not match kernel implementation Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba uhci stmf_sbd stmf zfs mm lofs idm mpt_sas sata random cpc logindmux ptm sppp nfs sd ipc ] > ::status debugging crash dump /manta/thoth/stor/thoth/8109ce59fb22301cf6d76158ae080f29/vmcore.0 (64-bit) from dal-mappblx001-17 operating system: 5.11 joyent_20160303T211841Z (i86pc) image uuid: (not set) panic message: NMI received dump content: kernel pages only > > !uname -v joyent_20170202T033902Z > > !mget -o ~/genunix.so ~~/stor/dmod-genunix28.so ~~/stor/dmod-genunix28.so [==============================================================================>] 100% 924.27KB > !mount -F lofs ~/genunix.so /usr/lib/mdb/kvm/amd64/genunix.so > ::unload genunix > ::load genunix > > ::ps S PID PPID PGID SID UID FLAGS ADDR NAME R 0 0 0 0 0 0x00000001 fffffffffbc304c0 sched R 90 0 0 0 0 0x00020001 ffffff4320466008 zpool-zones R 3 0 0 0 0 0x00020001 ffffff431d409020 fsflush R 2 0 0 0 0 0x00020001 ffffff431d40d018 pageout R 1 0 0 0 0 0x4a004000 ffffff431d009010 init R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010 smartlogin R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 ctrun R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 node R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020 zoneevent R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 node R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 ctrun R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 node R 42606 42574 8 8 0 0x4a004000 ffffff433266a000 zoneevent R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 ctrun R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 node R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 vmadm R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028 zoneadm R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040 zoneevent R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 ctrun R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 node R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 ctrun R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 node R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 node > > ::ps -f S PID PPID PGID SID UID FLAGS ADDR NAME R 0 0 0 0 0 0x00000001 fffffffffbc304c0 sched R 90 0 0 0 0 0x00020001 ffffff4320466008 zpool-zones R 3 0 0 0 0 0x00020001 ffffff431d409020 fsflush R 2 0 0 0 0 0x00020001 ffffff431d40d018 pageout R 1 0 0 0 0 0x4a004000 ffffff431d009010 /sbin/init R 95020 1 8 8 0 0x4a004000 ffffff4bf98ae010 /opt/smartdc/agents/bin/smartlogin-agent R 43138 1 8 8 0 0x4a004000 ffffff79d0840020 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/cn-age R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 /opt/smartdc/agents/lib/node_modules/cn-agent/node/bin/node --abort_on_uncaught R 43158 43139 8 8 0 0x4a004000 ffffff7adf68d020 /usr/vm/sbin/zoneevent R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 /opt/smartdc/agents/lib/node_modules/config-agent/build/node/bin/node /opt/smar R 42573 1 8 8 0 0x4a004000 ffffff79d0922028 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/vm-age R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 /opt/smartdc/agents/lib/node_modules/vm-agent/node/bin/node --abort_on_uncaught R 42606 42574 8 8 0 0x4a004000 ffffff433266a000 /usr/vm/sbin/zoneevent R 41316 1 8 8 0 0x4a004000 ffffffaafac7d008 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/net-ag R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 /opt/smartdc/agents/lib/node_modules/net-agent/node/bin/node --abort_on_uncaugh R 27788 41317 8 8 0 0x4a014000 ffffff6d138d3030 /usr/node/bin/node --abort_on_uncaught_exception /usr/sbin/vmadm lookup -j -o u R 27797 27788 8 8 0 0x4a004000 ffffff5e1c1b2028 /usr/sbin/zoneadm list -p -c R 41553 41317 8 8 0 0x4a004000 ffffffefd78a1040 /usr/vm/sbin/zoneevent R 40094 1 8 8 0 0x4a004000 ffffff431dc61038 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/hagfis R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 /opt/smartdc/agents/lib/node_modules/hagfish-watcher/build/node/bin/node /opt/s R 39501 1 39482 39482 0 0x4a004000 ffffff7cc447c040 /usr/bin/ctrun -l child -o noorphan /opt/smartdc/agents/lib/node_modules/firewa R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 /opt/smartdc/agents/lib/node_modules/firewaller/node/bin/node --abort_on_uncaug R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 /opt/smartdc/agents/lib/node_modules/cabase/cmd/node cmd/cainstsvc.js /opt/smar > > ::pgrep node S PID PPID PGID SID UID FLAGS ADDR NAME R 43139 43138 8 8 0 0x4a014000 ffffffaafa3f4048 node R 42863 1 42588 42588 0 0x4a014000 ffffff68dea0f058 node R 42574 42573 8 8 0 0x4a014000 ffffff444ab23020 node R 41317 41316 8 8 0 0x4a014000 ffffff4444147050 node R 40098 40094 8 8 0 0x4a014000 ffffff7cbada3020 node R 39502 39501 39482 39482 0 0x4a014000 ffffff7c5675c008 node R 37550 1 37508 37508 0 0x4a014000 ffffff431e166020 node R 36214 36206 8 8 0 0x4a014000 ffffff4331a5e010 node R 35525 35521 8 8 0 0x4a014000 ffffff8be3b32018 node R 22212 24025 22212 22212 0 0x4a014000 ffffff6d6b74c018 node R 18127 24045 24045 24045 9416 0x4a014000 ffffff5bc7724040 node R 18122 24045 24045 24045 9416 0x4a014000 ffffffb55c9d0048 node R 18116 24045 24045 24045 9416 0x4a014000 ffffff5bb5a22040 node R 18111 24045 24045 24045 9416 0x4a014000 ffffff5bb85bb038 node R 18102 24045 24045 24045 9416 0x4a014000 ffffff5bb9a8e000 node R 18101 24045 24045 24045 9416 0x4a014000 ffffffb588a16020 node R 18085 24045 24045 24045 9416 0x4a014000 ffffff5b934bf038 node R 18069 24045 24045 24045 9416 0x4a014000 ffffff5bcd892020 node R 18040 24045 24045 24045 9416 0x4a014000 ffffff5bdeaee048 node R 80388 50936 50936 50936 9415 0x4a014000 ffffff4454636040 node R 60532 80388 50936 50936 9415 0x4a014000 ffffff68dea0e000 node R 60524 80388 50936 50936 9415 0x4a014000 ffffff433191c040 node R 60519 80388 50936 50936 9415 0x4a014000 ffffffffa688c038 node > > ::ptree fffffffffbc304c0 sched ffffff4320466008 zpool-zones ffffff431d409020 fsflush ffffff431d40d018 pageout ffffff431d009010 init ffffff4bf98ae010 smartlogin ffffff79d0840020 ctrun ffffffaafa3f4048 node ffffff7adf68d020 zoneevent ffffff68dea0f058 node ffffff79d0922028 ctrun ffffff444ab23020 node ffffff433266a000 zoneevent ffffffaafac7d008 ctrun ffffff4444147050 node ffffff6d138d3030 vmadm ffffff5e1c1b2028 zoneadm ffffffefd78a1040 zoneevent ffffff431dc61038 ctrun ffffff7cbada3020 node ffffff7cc447c040 ctrun ffffff7c5675c008 node ffffff431e166020 node ffffff9c75aff040 ctrun > > ::walk proc | ::print proc_t p_dtrace_count p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 p_dtrace_count = 0 > > ::cpuinfo ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC 0 fffffffffbc326c0 1f 2 0 60 no no t-2 ffffffaaec8480c0 node 1 ffffff42ec7bc040 1f 1 0 169 no no t-8 ffffff01e8443c40 sched 2 fffffffffbc3cfc0 1b 1 0 60 no no t-3 ffffff4445ed2c40 node 3 ffffff42ec7a5500 1f 1 0 169 no no t-6 ffffff01e85a0c40 sched 4 ffffff42ec963540 1f 1 0 169 no no t-3 ffffff01e86e6c40 sched 5 ffffff42ec962040 1f 0 0 161 no no t-7 ffffff01e876fc40 sched 6 ffffff42ec960b00 1f 1 0 169 no no t-3 ffffff01e87efc40 sched 7 ffffff42ec95f500 1f 2 0 169 no no t-3 ffffff01e886fc40 sched 8 ffffff42ec95e000 1f 3 0 169 no no t-3 ffffff01e88fec40 sched 9 ffffff42ec958ac0 1f 1 0 169 no no t-3 ffffff01e899fc40 sched 10 ffffff42ec954b00 1f 11 0 169 no no t-3 ffffff01e8a2ec40 sched 11 ffffff42ec94ea80 1f 3 0 168 no no t-3 ffffff01e8bf7c40 sched 12 ffffff42eca63540 1f 14 0 168 no no t-3 ffffff01e8cd9c40 sched 13 ffffff42eca62040 1f 7 0 168 no no t-3 ffffff01e8d50c40 sched 14 ffffff42eca60080 1f 4 0 169 no no t-6 ffffff01e8df1c40 sched 15 ffffff42eca5a000 1f 2 0 169 no no t-3 ffffff01e8fc7c40 sched 16 ffffff42ecb2c080 1f 3 0 161 no no t-8 ffffff01e92d4c40 sched 17 ffffff42ecb2aa80 1f 2 0 169 no no t-3 ffffff01e934bc40 sched 18 ffffff42ecb23540 1f 1 0 165 no no t-3 ffffff01e93d4c40 sched 19 ffffff42ecb22040 1f 3 0 165 no no t-3 ffffff01e946fc40 sched 20 ffffff42ecb1eb00 1f 0 0 165 no no t-3 ffffff01e94f8c40 sched 21 ffffff430ce71500 1f 0 0 165 no no t-0 ffffff01e9566c40 sched 22 ffffff430ce69540 1f 0 0 165 no no t-7 ffffff01e95fec40 sched > > ffffff431e166020::pfiles FD TYPE VNODE INFO 0 CHR ffffff431d5a6b80 /devices/pseudo/mm@0:null 1 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log 2 REG ffffff4333461e00 /var/svc/log/smartdc-agent-ca-cainstsvc:default.log 3 PORT ffffff7c4d970780 [event port (port=ffffff96eea45a40)] 4 FIFO ffffffffbafb1e40 5 FIFO ffffff444ad1f300 6 REG ffffffaae317c380 /opt/smartdc/agents/log/ca/cainstsvc/amqp-2016-10-19.18:03:53.37508 7 SOCK ffffffaae1584800 socket: AF_INET 192.168.2.167 63975 remote: AF_INET 192.168.2.23 5672 > > ffffff431e166020::fd 0 ffffff4331bcb508 > > ::lminfo ADDR TP FLAG PID COMM VNODE PATH ffffff44424f1340 WR 0001 56600 zabbix_agentd ffffffaaf976c000 /zones/16c59be6-8e1 ffffff68de0c6180 WR 0021 51450 sac ffffffabf9611040 /zones/4a2d66d6-091 ffffffb116282980 WR 0021 24543 sac ffffffac0477c080 /zones/01fdb772-fea ffffff4445f2cb00 WR 0021 8084 sac ffffff444c21c100 /zones/1415a57e-e3b ffffff4e4cc6d240 WR 0021 92154 ttymon ffffffaad99a4100 /zones/d50bbf1d-6d3 ffffffa4b3e57580 WR 0021 23085 sac ffffffac00f56140 /zones/68fadf2d-fb8 ffffff444a863840 WR 0001 8046 zabbix_agentd ffffff4444491180 /zones/1415a57e-e3b ffffff4449bc6440 WR 0021 9025 sac ffffff444c93e280 /zones/c389d77f-ef7 ffffff444389cc40 WR 0021 7769 sac ffffff4441105300 /zones/16c59be6-8e1 ffffff444930f080 WR 0021 8903 ttymon ffffff4449e37340 /zones/2270bbde-8ae ffffff6d35f7fd40 WR 0001 25464 zabbix_agentd ffffff4461190340 /zones/68fadf2d-fb8 ffffff4333031a80 WR 0001 3059 sendmail ffffff4442c61380 /var/spool/clientmq ffffff443f580540 WR 0021 91301 sac ffffffaad9e01380 /zones/ee2cddf7-291 ffffff4443547980 WR 0001 3116 sendmail ffffff444420f400 /var/run/sendmail.p ffffff443f315340 WR 0021 7881 sac ffffff4444bb9440 /zones/7167d6f1-554 ffffffffbefbc740 WR 0021 61870 sac ffffffb84b39a400 /zones/8d230657-048 ffffffb11624ad80 WR 0021 154 sac ffffffaae892a400 /zones/c1ada27d-2d4 ffffff431dc7cd80 WR 0001 218 devfsadm ffffff43313ae540 /etc/dev/.devfsadm_ ffffff4447676600 WR 0021 8883 sac ffffff44443c7580 /zones/2270bbde-8ae ffffff4446604680 WR 0021 9036 ttymon ffffff444ba2e580 /zones/c389d77f-ef7 ffffffa4a9f9b280 WR 0001 64199 zabbix_agentd ffffffffb3c60580 /zones/f65d52b1-137 ffffff44462e5440 WR 0021 92150 sac ffffff4f1a42f680 /zones/d50bbf1d-6d3 ffffff4449bbe400 WR 0021 7782 ttymon ffffff4445bbe700 /zones/16c59be6-8e1 > > ::walk task_cache ! head -1 0xffffff4480d113c8 > 0xffffff4480d113c8::walk task 0xffffff6d6b74c018 > > ::walk as_cache ! head -1 0xffffffaad9b732e0 > 0xffffffaad9b732e0::as2proc ffffff444b845058