Segmentation fault when calling PyObject_CallObject() the second time

Question

I am writing a c++ class calling a python script. But when I call the method a second it stops working. I googled everywhere and finds no clue. I thought it is because of ref count problem, so I removed all the Py_DECREF call, but still no help.

Here is the c++ code (In the original code, I call PyObject_GetAttrString every time in RunModel, but in the second call gdb indicates the second func is <Py_NoneStruct>. So I changed to this):

#include <Python.h>
#include <fstream>
#include <iostream>
#include <stdexcept>
#include <thread>

using namespace std;

class PyRecognitionContext {

  PyObject *module, *func, *result, *args;

  public:

  PyRecognitionContext(int argc, char **argv) {
    Py_Initialize();
    PySys_SetArgv(argc, argv);
    module = PyImport_Import(PyString_FromString("model.t"));
    if (module == NULL) {
      PyErr_Print();
      throw std::invalid_argument("fails to import the module");
    }
  }

  ~PyRecognitionContext() {
    Py_Finalize();
  }

  void LoadModel(std::string model) {
    func = PyObject_GetAttrString(module, "load");
    args = PyTuple_New(1);
    PyTuple_SetItem(args, 0, PyString_FromString(model.c_str()));
    PyObject_CallObject(func, args);
    func = PyObject_GetAttrString(module, "run");
  }

  char* RunModel(const char *imageBytes, size_t size) {
    args = PyTuple_New(1);
    PyTuple_SetItem(args, 0, PyByteArray_FromStringAndSize(imageBytes, size));
    result = PyObject_CallObject(func, args);
    char* resultStr = PyString_AsString(result);
    return resultStr;
  }

};

int task(PyRecognitionContext pr, string fileName) {
  cout << "//";
  cout << fileName << endl;
  std::ifstream ifs(fileName, ios::binary|ios::ate);
  size_t imgSize = ifs.tellg();
  char *result = new char[imgSize];
  ifs.seekg(0, ios::beg);
  ifs.read(result, imgSize);
  ifs.close();
  std::cout << pr.RunModel(result, imgSize) << endl;
  return 0;
}

int main(int argc, char *argv[]) {
  PyRecognitionContext pr(argc, argv);
  pr.LoadModel("/home/dickzhou/data/sex_256/cp/");
  cout << "//loaded" << endl;

  task(pr, "/home/dickzhou/data/face/male/24-28/24/135.JPG");
  task(pr, "/home/dickzhou/data/face/female/24-28/24/138.JPG");

  //thread t1(task, pr, "/home/dickzhou/data/face/male/24-28/24/135.JPG");
  //thread t2(task, pr, "/home/dickzhou/data/face/female/24-28/24/125.JPG");

  //t1.join();
  //t2.join();

  cout << "//done" << endl;
  return 0;
}

I compiled it using:

g++ -I/usr/include/python2.7 -I/usr/include/python2.7 -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O1 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -lpthread -ldl -lutil -lm -lpython2.7 -Xlinker -export-dynamic -std=c++11 pdr.cpp

and this is my gdb output:

(gdb) break 40
Breakpoint 1 at 0x401f47: file pdr.cpp, line 40.
(gdb) run
Starting program: /home/dickzhou/data/sex_256/c++/a.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
//loaded
///home/dickzhou/data/face/male/24-28/24/135.JPG

Breakpoint 1, RunModel (size=14637, imageBytes=0x6e5270 "\377\330\377", <incomplete sequence \340>, this=0x7fffffffe040) at pdr.cpp:40
40          result = PyObject_CallObject(func, args);
Missing separate debuginfos, use: debuginfo-install glibc-2.17-105.el7.x86_64 libgcc-4.8.5-16.el7.x86_64 libstdc++-4.8.5-4.el7.x86_64 python-libs-2.7.5-34.el7.x86_64
(gdb) p func
$1 = (PyObject *) 0x2aaaaabfa9b0
(gdb) p args
$2 = (PyObject *) 0x2aaaaabf2d10
(gdb) c
Continuing.
a
///home/dickzhou/data/face/female/24-28/24/138.JPG

Breakpoint 1, RunModel (size=13572, imageBytes=0x68d9f0 "\377\330\377", <incomplete sequence \340>, this=0x7fffffffe040) at pdr.cpp:40
40          result = PyObject_CallObject(func, args);
(gdb) p func
$3 = (PyObject *) 0x2aaaaabfa9b0
(gdb) p args
$4 = (PyObject *) 0x2aaaaab8bbd0
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00002aaaab33c09a in PyObject_Call () from /lib64/libpython2.7.so.1.0
(gdb)

a minimal python code would be:

def load(str):
    pass

def run(bytes):
    return "a"

You're **never** checking the return values from the Python functions. You must **always** check the return values, and act upon them! — Antti Haapala -- Слава Україні, Jan 07 '18 at 13:48
@AnttiHaapala I was just quickly prototyping, and will definitely clean up this mess. I think it is not related to the problem, right? — DickRD, Jan 07 '18 at 14:03
It could very much **be**, but we wouldn't know now would we!?! — Antti Haapala -- Слава Україні, Jan 07 '18 at 15:34
Anyway, `Py_NoneStruct` is the implementation object of `None`, so your function is `None` and `None` is not callable, so the `PyObject_CallObject` would return `NULL`. — Antti Haapala -- Слава Україні, Jan 07 '18 at 15:36
@AnttiHaapala Thanks for your help. But in the code I posted, `func` is not `None`, and you can see it in the gdb output. It is the same value as last call. I even check just now to be sure, and `PyCallable_Check(func)` returned `1` which I think means it can be called. — DickRD, Jan 08 '18 at 00:48
The `Py_NoneStruct`, however, shows up the second time I call function `RunModel`, if I move the line `func = PyObject_GetAttrString(module, "run");` to the beginning of `RunModel` function. I don't know why. — DickRD, Jan 08 '18 at 00:50

score 2 · Answer 1 · answered Jan 08 '18 at 04:01

Your actual problem is not following Rule of Zero. You have a class with resources (a resource is anything that needs freeing later) managed manually. That won't do.

Instances of PyRecognitionContext are copied, and Py_Finalize() is called twice, and also prematurely.

The global Python context needs initialization and finalization. We can delegate that to a separate class to facilitate Single Responsibility Principle.

struct PyContext
{
    PyContext()
    {
        Py_Initialize();
    }

    ~PyContext()
    {
        Py_Finalize();
    }

    PyContext& operator=(const PyContext&) = delete;
    PyContext(const PyContext&) = delete;
    PyContext& operator=(PyContext&&) = delete;
    PyContext(PyContext&&) = delete;
};

Now you can put a member of that class in your PyRecognitionContext

class PyRecognitionContext {
    PyContext pyContext;

    // ... rest of the code
};

This moves the run time error to the compile time.

The other Python objects are resources too because they need to be freed. We can use std::unique_ptr with a custom deleter to manage these.

score 0 · Answer 2 · answered Jan 08 '18 at 02:09

Thanks for all your help, I have solved this question.

The problem was the deconstruct function being called twice because the task function is pass-by-value. I changed it to pass-by-reference:

int task(PyRecognitionContext *pr, string fileName) {
  cout << "//";
  cout << fileName << endl;
  std::ifstream ifs(fileName.c_str(), ios::binary|ios::ate);
  size_t imgSize = ifs.tellg();
  char *result = new char[imgSize];
  ifs.seekg(0, ios::beg);
  ifs.read(result, imgSize);
  ifs.close();
  std::cout << pr->RunModel(result, imgSize) << endl;
  return 0;
}

You're leaking memory,too, the allocated bytestring is never released. — Antti Haapala -- Слава Україні, Jan 08 '18 at 03:48

Segmentation fault when calling PyObject_CallObject() the second time

2 Answers2