Bug #16272


python support in pstack should work again

Added by Bill Sommerfeld 11 days ago. Updated 3 days ago.

cmd - userland programs
Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:
External Bug:


The pstack command has support for using the python debug helper to extract python source filename, line number, and function name out of the python interpreter.

Unfortunately, some of the necessary infrastructure for this hasn't evolved as python has evolved, and it stopped working some time ago and nobody noticed.

Under the covers, pstack looks to see if the target process has loaded; if it does, pstack attempts to load, and looks for pydb_pc_frameinfo, and if found, calls it with an assortment of parameters for each stack frame in the traceback. If it returns success, it prints out the string produced by the helper library.

I've gone spelunking into the code and got it to work again - see for sample output.
I found I needed to make the following changes:

  1. python needs to be built with CTF information and -msave-args as that's what pstack uses to infer argument count and actually extract function arguments out of the process address space.
  2. The python library needs to look for the proper symbol (it was looking for PyEval_EvalFrameEx, but in 3.9, the function of interest is named _PyEval_EvalFrameDefault
  3. pstack needs to pass pydb_pc_frameinfo the second argument rather than the first, as that's where the new function takes the frame parameter.

Steps 1 and 2 are outside of the scope of illumos-gate and will have to be fixed by each distribution. Step 3 needs to be fixed in illumos-gate.

I believe the right way to handle (3) is for pstack to look for a different entrypoint in libpython*_db and pass argv instead of argv[0] if it sees the new entrypoint (argv, not argv[1], so that we don't have to revise pstack again if they change the calling sequence).

There may also need to be some work to better handle the foo.cold symbols produced by profile-guided optimization (PGO moves some code to a separate .cold symbol), as openindiana's python build enables that feature.

Related issues

Related to illumos gate - Bug #16302: python support in pstack needs further updates for python versions 3.11 and higher.NewBill Sommerfeld

Actions #1

Updated by Electric Monk 9 days ago

  • Gerrit CR set to 3290
Actions #2

Updated by Bill Sommerfeld 9 days ago

  • Status changed from New to In Progress
Actions #3

Updated by Bill Sommerfeld 9 days ago

The Python side of this change is available at

Actions #4

Updated by Bill Sommerfeld 6 days ago

I'm now digging into what changes will be required for newer versions of Python (specifically, 3.12 as I'm working on packaging this for openindiana).

Until recently, cpython's main interpreter function with each call in python requiring a C call to the interpreter loop; calls to other functions turned into calls to C functions that eventually made it back to another call to the interpreter.

This was convenient for the python plugin for pstack - one C frame results in at most one python frame.

But in 3.11, they changed it up so that python calls can happen inside the interpreter loop without making a recursive call to the interpreter. So a single C frame can result in multiple Python frames in the trace. And additional changes to the pystack-to-helper interface are needed. You can find the chain of frames linked from the PyThreadState, which points to a linked list of _PyCFrame structures (which are stack allocated); the _PyCFrame also links to the _PyInterpreterFrame which is 1:1 with python invocations. Finding the relevant _PyCFrame from the chain in the PyThreadState requires access to the C frame's address on the stack, so that will have to go over to the helper, too.

Some discussion of the data structures here:

Actions #5

Updated by Bill Sommerfeld 4 days ago

As what I have for python 3.9 is in good shape (and the equivalent for 3.12 requires a bit more thought) I'll limit this bug to fixing it for 3.9

Testing notes:

ONU'ed a nightly built onto an openindiana VM updated to include the python piece of this, repeatedly ran pstack on a running "pkg refresh", and got sane-looking python annotations in the output. I cross-checked some of the line numbers and found the expected call sites.

Sample output:

 fffffc7fef20a20a ioctl () + a
 fffffc7fea54cc80 zfs_iter_filesystems (145f0b0, fffffc7fea5b1c60, fffffc7fffdfd6d0) + 90
 fffffc7fea5b1e6d be_add_children_callback (145f0b0, fffffc7fffdfd6d0) + 20d
 fffffc7fea54cc9d zfs_iter_filesystems (1468fb0, fffffc7fea5b1c60, fffffc7fffdfd6d0) + ad
 fffffc7fea5b1abc be_get_list_callback (1462100, fffffc7fffdfd6d0) + 16c
 fffffc7fea53bd3d zpool_iter (145c600, fffffc7fea5b1950, fffffc7fffdfd6d0) + 9d
 fffffc7fea5b1591 _be_list (0, fffffc7fffdfdcf8, 0) + 1a1
 fffffc7fea5b12eb be_list (0, fffffc7fffdfdcf8, 0) + 5b
 fffffc7fea5e3012 beList (fffffc7fea6095e0, fffffc7fef0b3040, fffffc7fe9641b80) + a2
 fffffc7fec880c3d cfunction_call (fffffc7fea6097c0, fffffc7fef0b3040, fffffc7fe9641b80) + 3d
 fffffc7fec846e9b _PyObject_MakeTpCall (41b580, fffffc7fea6097c0, fffffc7fe95ca528, 0, fffffc7fea6bdc70) + 10b
 fffffc7fec900864 _PyEval_EvalFrameDefault (41b580, fffffc7fe95ca3a0, 0) + 61d4
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (get_be_list) ]
 fffffc7fec84799b function_code_fastcall (41b580, fffffc7fea603a80, fffffc7fea6070e8, 1, fffffc7fea6b8a80) + ab
 fffffc7fec8ffea4 _PyEval_EvalFrameDefault (41b580, fffffc7fe9647210, 0) + 5814
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (get_be_name) ]
 fffffc7fec84799b function_code_fastcall (41b580, fffffc7fea603d40, 145c5a0, 1, fffffc7fea6b8a80) + ab
 fffffc7fec8ffea4 _PyEval_EvalFrameDefault (41b580, 145c3a0, 0) + 5814
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (refresh_publishers) ]
 fffffc7fec8f9749 _PyEval_EvalCode (41b580, fffffc7fe9dcc920, fffffc7fea605c00, 0, fffffc7fe9583ef0, 1, fffffc7fea69cf58, fffffc7fe9583ef8, ...) + 289
 fffffc7fec8477ec _PyFunction_Vectorcall (fffffc7fe9675ee0, fffffc7fe9583ef0, 1, fffffc7fea69cf40) + ec
 fffffc7fec849c8d method_vectorcall (fffffc7fe9585840, fffffc7fe9583ef8, 8000000000000000, fffffc7fea69cf40) + 7d
 fffffc7fec8fbb88 _PyEval_EvalFrameDefault (41b580, fffffc7fe9583d60, 0) + 14f8
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (__refresh) ]
 fffffc7fec8f9749 _PyEval_EvalCode (41b580, fffffc7fea6a07c0, fffffc7fee4d8400, 0, 1441d00, 1, fffffc7fea69ce18, 1441d08, ...) + 289
 fffffc7fec8477ec _PyFunction_Vectorcall (fffffc7fe9685790, 1441d00, 1, fffffc7fea69ce00) + ec
 fffffc7fec849c8d method_vectorcall (fffffc7fe996dcc0, 1441d08, 8000000000000000, fffffc7fea69ce00) + 7d
 fffffc7fec8fbb88 _PyEval_EvalFrameDefault (41b580, 1441b70, 0) + 14f8
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (refresh) ]
 fffffc7fec8f9749 _PyEval_EvalCode (41b580, fffffc7fea6a0710, fffffc7fee4d8400, 0, fffffc7fe957ef58, 1, fffffc7feecadb98, fffffc7fe957ef60, ...) + 289
 fffffc7fec8477ec _PyFunction_Vectorcall (fffffc7fe9685700, fffffc7fe957ef58, 1, fffffc7feecadb80) + ec
 fffffc7fec849c8d method_vectorcall (fffffc7fe9585080, fffffc7fe957ef60, 8000000000000000, fffffc7feecadb80) + 7d
 fffffc7fec8fbb88 _PyEval_EvalFrameDefault (41b580, fffffc7fe957edd0, 0) + 14f8
   [ /usr/bin/pkg:3880 (__refresh) ]
 fffffc7fec8f9749 _PyEval_EvalCode (41b580, fffffc7feed48660, fffffc7fef016300, 0, fffffc7fe9583b48, 2, fffffc7feed9fcb8, fffffc7fe9583b58, ...) + 289
 fffffc7fec8477ec _PyFunction_Vectorcall (fffffc7fe963ddc0, fffffc7fe9583b48, 8000000000000002, fffffc7feed9fca0) + ec
 fffffc7fec8fbb88 _PyEval_EvalFrameDefault (41b580, fffffc7fe95839a0, 0) + 14f8
   [ /usr/bin/pkg:3917 (publisher_refresh) ]
 fffffc7fec84799b function_code_fastcall (41b580, fffffc7feed487c0, 54fb58, 2, fffffc7fef016300) + ab
 fffffc7fec8fab7f _PyEval_EvalFrameDefault (41b580, 54f920, 0) + 4ef
   [ /usr/bin/pkg:5864 (main_func) ]
 fffffc7fec84799b function_code_fastcall (41b580, fffffc7feed39ea0, fffffc7fef0b3058, 0, fffffc7fef016300) + ab
 fffffc7fec8fedda _PyEval_EvalFrameDefault (41b580, 143e000, 0) + 474a
   [ /usr/bin/pkg:5962 (handle_errors) ]
 fffffc7fec8f9749 _PyEval_EvalCode (41b580, fffffc7feeda70e0, fffffc7fef016300, 0, 7805d0, 1, 0, 7805d8, ...) + 289
 fffffc7fec8477ec _PyFunction_Vectorcall (fffffc7fe963f280, 7805d0, 8000000000000001, 0) + ec
 fffffc7fec8fab7f _PyEval_EvalFrameDefault (41b580, 780460, 0) + 4ef
   [ /usr/bin/pkg:6405 (<module>) ]
 fffffc7fec8f9749 _PyEval_EvalCode (41b580, fffffc7feeda72f0, fffffc7fef016300, fffffc7fef016300, 0, 0, 0, 0, ...) + 289
 fffffc7fec8f94b1 _PyEval_EvalCodeWithName (fffffc7feeda72f0, fffffc7fef016300, fffffc7fef016300, 0, 0, 0, 0, 0, ...) + 51
 fffffc7fec8f944f PyEval_EvalCodeEx (fffffc7feeda72f0, fffffc7fef016300, fffffc7fef016300, 0, 0, 0, 0, 0, ...) + 3f
 fffffc7fec8f9402 PyEval_EvalCode (fffffc7feeda72f0, fffffc7fef016300, fffffc7fef016300) + 22
 fffffc7fec936e16 run_eval_code_obj (41b580, fffffc7feeda72f0, fffffc7fef016300, fffffc7fef016300) + 56
 fffffc7fec936d8d run_mod (ebee00, fffffc7fef01e830, fffffc7fef016300, fffffc7fef016300, fffffc7fffdffb58, fffffc7fef0e64b0) + 6d
 fffffc7fec937ab4 pyrun_file (fffffc7fef27b0a0, fffffc7fef01e830, 101, fffffc7fef016300, fffffc7fef016300, 1, fffffc7fffdffb58) + a4
 fffffc7fec9376a0 PyRun_SimpleFileExFlags (fffffc7fef27b0a0, fffffc7fef097b30, 1, fffffc7fffdffb58) + 1b0
 fffffc7fec952a63 Py_RunMain () + 2a3
 fffffc7fec95263b Py_BytesMain (4, fffffc7fffdffca8) + 2b
 0000000000400fd7 _start_crt () + 87
 0000000000400f38 _start () + 18

Sample output filtered to just the Python frames:

openindiana# pstack $(pgrep pkg) | grep '^   \['
   [ /usr/lib/python3.9/vendor-packages/pkg/ (add) ]
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (__rebuild_catalog) ]
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (__refresh) ]
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (refresh) ]
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (refresh_publishers) ]
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (__refresh) ]
   [ /usr/lib/python3.9/vendor-packages/pkg/client/ (refresh) ]
   [ /usr/bin/pkg:3880 (__refresh) ]
   [ /usr/bin/pkg:3917 (publisher_refresh) ]
   [ /usr/bin/pkg:5864 (main_func) ]
   [ /usr/bin/pkg:5962 (handle_errors) ]
   [ /usr/bin/pkg:6405 (<module>) ]
Actions #6

Updated by Bill Sommerfeld 4 days ago

  • Related to Bug #16302: python support in pstack needs further updates for python versions 3.11 and higher. added
Actions #7

Updated by Bill Sommerfeld 4 days ago

  • Status changed from In Progress to Pending RTI
Actions #8

Updated by Bill Sommerfeld 3 days ago

The interface introduced here is likely to be able to work with Python 3.10 but I haven't attempted to port the libpythonXY_db.c code to 3.10 and there could be other showstoppers.

3.10 support is not likely that interesting as Openindiana is working on moving forward to 3.12 (skipping 3.10 and 3.11), while Omnios has been on 3.11 for a while now.

Actions #9

Updated by Electric Monk 3 days ago

  • Status changed from Pending RTI to Closed
  • % Done changed from 0 to 100

git commit 616d60f414d125f2dfc25908ca90a7c4451076c6

commit  616d60f414d125f2dfc25908ca90a7c4451076c6
Author: Bill Sommerfeld <>
Date:   2024-02-19T16:04:31.000Z

    16272 python support in pstack should work again
    Reviewed by: Marcel Telka <>
    Approved by: Dan McDonald <>


Also available in: Atom PDF