How to convert a 'managedbuffer' into a Callable in Python

Question

I have written a Python Wrapper for a C++14 library using SWIG. Within the C++ API I can register std::functions as callbacks.

I have a SWIG typemap for std::function's to pass a lambda expression which invokes the Python callback:

%typemap(in) std::function {
    auto callback = [$input](auto&&... params) {
       PyGILState_STATE state = PyGILState_Ensure();

       PyObject* result =  PyObject_CallFunctionObjArgs($input,makePyObject(std::forward<decltype(params)>(params))..., NULL);
       const int retVal = PyObject_IsTrue(result);

       Py_DECREF(result);
       PyGILState_Release(state);
       return retVal == 1;
   };
   $1 = std::move(callback);
}

When I run a test script, the following Python expression works fine:

callback = lambda a,b: self.doStuff(a,b)
self.cppInterface.registerFunc(callback)

This expression however does not work:

self.cppInterface.registerFunc(lambda a,b: self.doStuff)

When I pass the lambda directly to the register function, I get a the following error when the callback is called from C++:

TypeError: 'managedbuffer' object is not callable

Why is the PyObject $input not a callable? How do I allow both Python expressions?

Example code:

https://github.com/nullmedium/python-swig-demo

What does 'does not work' mean? You get a TypeError from the typemap? What typemap are you using for this? (Can you show that in an MCVE?) — Flexo, Jun 07 '18 at 16:10
I get the following error: TypeError: 'managedbuffer' object is not callable — Jens Luedicke, Jun 08 '18 at 13:36
My suspicion is that Python handles the two expressions very differently and the $input PyObject must be processed differently. But I do not know how to get from a managedbuffer Type to a callable function. — Jens Luedicke, Jun 08 '18 at 13:46
My suspicion is that there's more to this than meets the eye, likely a reference counting problem with the ownership of `$input`. The only obvious difference here is in the lifetime of the two Python lambda objects. So you need a way to call `Py_INCREF($input);` and `Py_DECREF($input);` when the lambda is destroyed, but I can't prove that from what you've shown here. (Which honestly is kinda frustrating) — Flexo, Jun 08 '18 at 18:58
I just tried to recreate a simple example (at home) but on MacOS the problems are kind of different. The callback = lambda a,b: self.doStuff(a,b) expression does not work, because doStuff is not found in the self dict. — Jens Luedicke, Jun 10 '18 at 11:22

score 1 · Accepted Answer · edited Jul 04 '18 at 18:22

Looks like you have a reference counting problem. You need to retain a reference to $input, even after the std::function typemap has completed. Otherwise it's going to be missing a reference once the call to registerFunc is done. The simplest way to do this is to make your typemap capture a std::shared_ptr instead of the raw PyObject, e.g.:

%typemap(in) std::function {
    Py_INCREF($input);
    static const auto decref = [](PyObject *o) {
        Py_DECREF(o); // This needs to be another lambda/function because Py_DECREF is really a macro
    };
    std::shared_ptr<PyObject> callable($input, decref);
    auto callback = [callable](auto&&... params) {
       PyGILState_STATE state = PyGILState_Ensure();
                                                      // Back to raw PyObject    
       PyObject* result =  PyObject_CallFunctionObjArgs(callable.get(),makePyObject(std::forward<decltype(params)>(params))..., NULL);
       int retVal = -1;

       if (result)
       {
           retVal = PyObject_IsTrue(result);
           Py_DECREF(result);
       }

       PyGILState_Release(state);
       return retVal == 1;
   };
   $1 = std::move(callback);
}

I would have used a std::unique_ptr instead ideally, but even with generalised lambda captures that stops copy construction from being possible and forcing SWIG to generate code without needing that is a bit more work.

I'd probably throw at least a PyCallable_Check in the typemap for good measure though too.

Thanks! This indeed solves the problem. I think I will create a RAII struct to call Py_INCREF in the c'tor and Py_DECREF in the d'tor and pass this to the lambda. I have made one for the GIL state already. — Jens Luedicke, Jun 11 '18 at 08:51

How to convert a 'managedbuffer' into a Callable in Python

1 Answers1