Python to C to Python call flow causing Segmentation fault

Question

I have a requirement to call C function from python, and that C function in turn invokes python methods. Ex: test.py invokes C function in hello.c, and that C function calls method in hello.py. In this case, when I try to load .py module (using PyImport_ImportModule) in C function, causing Segmentation fault (core dumped).
Can you please help debug/root cause this issue? Let me know, if I am missing anything or is there any other way to get this working?

test.py is as follows which uses ctypes and invokes C function.

import time
import ctypes

def _run():
    _hello = ctypes.CDLL('./hello.so')
    result = _hello.hello_main()
    time.sleep(10)

if __name__ == "__main__" :
    _run()

hello.c is as follows, which is loading hello py module.

#include <Python.h>

int hello_main()
{
     PyObject *pModule = NULL;
     PyObject *pFunc   = NULL;

     Py_Initialize();
     printf("Import Py module\n");
     pModule = PyImport_ImportModule("hello");
     if (pModule == NULL) {
         printf("Py module import fail\n");
         return 0;
     }

     pFunc = PyObject_GetAttrString(pModule, "Hello");
     if (pFunc == NULL) {
         printf("Py module get attr fail \n");
         return 0;
     }
     PyEval_CallObject(pFunc, NULL);

     Py_Finalize();

     return 0;
}

hello.py is as follows -

#Filename hello.py
def Hello():
    print "Hello, Py world!"

The C code is compiled as shared lib as below:

[host]# gcc -g -o hello.so hello.c -shared -fPIC -I/usr/include/python2.7 -L/usr/lib/python2.7 -lpython2.7

and i get Segmentation fault, when i run test.py

[host]# python test.py 
Import Py module
Segmentation fault (core dumped)

Following is the backtrace of the core:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7b116a0 in threadstate_getframe () from /lib64/libpython2.7.so.1.0
(gdb) bt
#0  0x00007ffff7b116a0 in threadstate_getframe ()
   from /lib64/libpython2.7.so.1.0
#1  0x00007ffff7aef5c9 in PyEval_GetGlobals () from /lib64/libpython2.7.so.1.0
#2  0x00007ffff7b0b222 in PyImport_Import () from /lib64/libpython2.7.so.1.0
#3  0x00007ffff7b0b48a in PyImport_ImportModule ()
   from /lib64/libpython2.7.so.1.0
#4  0x00007fffefdcc980 in hello_main () at hello.c:15
#5  0x00007ffff01dcdcc in ffi_call_unix64 () from /lib64/libffi.so.6
#6  0x00007ffff01dc6f5 in ffi_call () from /lib64/libffi.so.6
#7  0x00007ffff03efc8b in _ctypes_callproc ()
   from /usr/lib64/python2.7/lib-dynload/_ctypes.so
#8  0x00007ffff03e9a85 in PyCFuncPtr_call ()
   from /usr/lib64/python2.7/lib-dynload/_ctypes.so
#9  0x00007ffff7a5d8e3 in PyObject_Call () from /lib64/libpython2.7.so.1.0
#10 0x00007ffff7af2036 in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#11 0x00007ffff7af64bd in PyEval_EvalFrameEx () from /lib64/libpython2.7.so.1.0
#12 0x00007ffff7af8e3d in PyEval_EvalCodeEx () from /lib64/libpython2.7.so.1.0
#13 0x00007ffff7af8f42 in PyEval_EvalCode () from /lib64/libpython2.7.so.1.0
#14 0x00007ffff7b1237f in run_mod () from /lib64/libpython2.7.so.1.0
#15 0x00007ffff7b1353e in PyRun_FileExFlags () from /lib64/libpython2.7.so.1.0
#16 0x00007ffff7b147c9 in PyRun_SimpleFileExFlags ()
   from /lib64/libpython2.7.so.1.0
#17 0x00007ffff7b2591f in Py_Main () from /lib64/libpython2.7.so.1.0
---Type <return> to continue, or q <return> to quit---
#18 0x00007ffff6d4db35 in __libc_start_main () from /lib64/libc.so.6
#19 0x0000000000400721 in _start ()
(gdb) quit

EDIT 1: The reason for the crash is that, there is no thread state in my example case, leading to crash in the call flow below. The tstate is NULL.

PyEval_GetGlobals(void)
 - PyEval_GetFrame();
     - PyThreadState *tstate = PyThreadState_GET();
     - return _PyThreadState_GetFrame(tstate);      ==> Crashing here

As explained in Python PyGILState_{Ensure/Release} causes segfault while returning to C++ from Python code

adding below code after Py_Initialize(), fix the crash problem.

PyGILState_STATE d_gstate;
d_gstate = PyGILState_Ensure();

Thanks for all the help.

Have you looked into the dumped core? Which function caused it, in which file, on which line? — DYZ, Apr 29 '17 at 05:46
The `CDLL` call doesn't hold the GIL. Then to add to this insanity you're trying to reinitialize the interpreter. Why can't you give your C function a ctypes callback function to call back into Python? ctypes will handle the details of reacquiring and releasing the GIL for you. — Eryk Sun, Apr 29 '17 at 11:57
Thanks for reply. The requirement is that - we have some feature code that was written C and python. The C provides the API's for that featue. Then, we have another Python application, where we want use that existing feature by calling C API's (which inturn invokes python module internally). We dont have much flexibility to rewrite/modify the exiting C API code, but just find glue to use the C APIs from python application. Is there way to make this working? Thanks a lot for help. — Laxmikantha Reddy Ponnuru, May 01 '17 at 03:32
It sounds to me like you need to wrap your pre-existing C API in a C extension. Functions in a C extension can be used to convert Python variables to C variables, make calls to the C API and return the value to Python. I've done just this for academia and for my occupation. Read more [here](https://docs.python.org/2/extending/extending.html). — JacaByte, May 02 '17 at 03:24
P.S. based on your comments here it sounds like you've fallen prey to the [XY Problem.](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem) If you still haven't found a satisfactory answer perhaps you should consider re-asking it, but about what you _really_ want instead of about why your attempted solution didn't work. — JacaByte, May 02 '17 at 03:27
IMHO, you should not have edited the question with the answer, but you should actually post an answer. Simply explain either in the answer itself or in a comment that comments helped you to find it. — Serge Ballesta, May 03 '17 at 07:36

Python to C to Python call flow causing Segmentation fault

0 Answers0