How to use a Cython cdef class member method in a native callback

Question

I have a C++ library that is heavily callback driven. The callbacks are registered as std::function instances.

A callback registration function might look something like:

void register(std::function<void(int)> callback);

I can register plain cdef functions by making libcpp.functional.function objects. Say I have this callback:

cdef void my_callback(int n):
    # do something interesting with n

I can register it succesfully:

cdef function[void(int)]* f_my_callback = new function[void(int)](my_callback)
register(deref(f_my_callback))

The problem is that I want to register cdef class methods as the callback. Let's say I have a class:

cdef class MyClass:

    cdef function[void(int)]* f_callback
    py_callback = lambda x: None

    # some more init/implementation

    def __init__(self):
        # ** this is where the problem is **
        self.f_callback = new function[void(int)](self.member_callback)
        register(deref(self.f_callback))

    cdef void member_callback(self, int n) with gil:
        self.py_callback(n)

    def register_py(self, py_callback):
        self.py_callback = py_callback

the self.f_callback = new function[void(int)](self.member_callback) line doesn't work because the function[T] constructor is seeing the MyClass parameter (self). In regular python, doing self.member_callback is basically equivalent to partially applying self to MyClass.member_callback, so I thought that this would be fine, but it is not.

I made a typedef:

ctypedef void (*member_type)(int)

Then if I cast, I can get it to compile:

self.f_callback = new function[void(int)](<member_type>self.member_callback)

But when callbacks actually come in, everything is broken. Doing anything with self results in a segfault.

What is the 'right' way to pass a cython class member method as a C/C++ callback?

EDIT:

Just to clarify, this question is specifically about passing member methods (i.e. function with self as the first argument) to C++ functions expecting a parameter of type std::function.

If I decorate the member_callback with @staticmethod and remove all references to self it all works as expected. Unfortunately, the method needs to have access to self to correctly execute the logic for the class instance.

Answer 1

std::function can take a range of arguments. In its simplest form it takes a function pointer that directly maps to its template type, and this is all that the Cython wrappers for it are really set up to cope with. For wrapping a C++ member function you'd typically use std::mem_fun or std::mem_fun_ref to either take a copy of, or a reference to, the relevant class instance (in modern C++ you might also choose a lambda function, but that really just gives you the same options).

Converting this to Python/Cython you need to hold a PyObject* to the object whose member you are calling, and with that comes a responsibility to handle the reference counting yourself. Unfortunately Cython can't currently generate the wrappers for you, and so you need to write a holder yourself. Essentially you need a callable object that handles the reference counting.

In a previous answer I showed two ways of creating this kind of wrapper (one of which was manual, and the second of which saved effort by using Boost Python, which has already implemented something very similar). The scheme I showed there will work for any Python callable matching the relevant signature. Although I illustrated it with a standard Python module level function it would work equally well for a member function since instance.memberfunction generates a bound member function - a Python callable which will work just as well.

For your problem the only difference is that you are using a cdef member function rather than a def member function. This looks like it should work but doesn't (recent versions of Cython attempt to do an automatic conversion to a Python callable but it fails with a runtime error). You can create a lambda function though as a very simple wrapper. This is slightly less-than-ideal from a speed point of view but has the advantage of working with the existing code.

The modifications you will need to make to earlier answer are as follows:

• If you are trying to use the manual PyObjectWrapper version then change the argument types to match your signature (i.e. change int, string& to int). You don't have to do this for the boost version.

• Your wrapper for void register(std::function<void(int)> callback); needs to be:

  cdef extern from "whatever.hpp":
    void register(PyObjWrapper)
    # or
    void register(bpo)
  

  depending on which version you're using. This is lying to Cython about the signature, but since both of these objects are callable C++ objects they are automatically convertable to std::function by the C++ compiler.

• Call register as register(PyObjWrapper(lambda x: self.member_callback(x))) or register(get_as_bpo(lambda x: self.member_callback(x))).

---

It would be possible to create a more efficient version specifically targeted at using cdef functions. It would be based around an object that is pretty similar to a PyObjWrapper. However, I'm reluctant to do this without a clear justification given that the code I've already written will work.

Answer 2

I believe that @DavidW's answer is correct for the simplified version of the problem that asked.

I presented a simplified version of my question in the interest of clarity, but so far as I can tell, the real version of my problem needs a slightly different approach.

Specifically, in my simplified version, I indicated that the callback would be called with an int. If this was the case, I could conceivably provide a python callable (either the boost python or the PyObjWrapper version).

However, the callbacks are actually called with a std::shared_ptr<T> where T is a templated class in the library. The cdef callback needs to do something with that instance and eventually call a python callable. So far as I can tell, there isn't a way to call a python function with a C++ object like a std::shared_ptr.

@DavidW's response was quite helpful though, and it led me to a solution that does work for this kind of situation. I made a C++ utility function:

// wrapper.hpp

#include <memory>
#include <functional>

#include "my_cpp_project.hpp"

template<typename T>
using cy_callback = void (*)(void*, std::shared_ptr<my_cpp_class<T>>);

template<typename T>
class function_wrapper {
public:

    static
    std::function<void(std::shared_ptr<my_cpp_class<T>>)>
    make_std_function(cy_callback<T> callback, void* c_self)
    {
        std::function<void(std::shared_ptr<my_cpp_class<T>>)>
        wrapper = [=](std::shared_ptr<my_cpp_class<T>> sample) -> void
        {
            callback(c_self, sample);
        };
        return wrapper;
    }

};

Which I wrapped in cython:

cdef extern from "wrapper.hpp":
    cdef cppclass function_wrapper[T]:

        ctypedef void (*cy_callback) (void*, shared_ptr[my_cpp_class[T]])

        @staticmethod
        function[void(shared_ptr[my_cpp_class[T]])] make_std_function(
            cy_callback, void*)

Basically, the idea is that I pass the cython class callback method to the wrapper and it returns a std::function version of it with the right type signature. The returned function is actually a lambda which applies the self argument to the callback.

Now I can make my cdef member methods and they get called correctly:

    cdef void member_callback(self, shared_ptr[my_cpp_class[specific_type]] sample) with gil:
        # do stuff with the sample ...
        self.python_cb(sample_related_stuff)

registration looks something like:

register(function_wrapper[specific_type].make_std_function(<cb_type>self.member_callback, <void*>self))

Note the cb_type cast in there. The wrapper is expecting a void* so we need to cast it to match. This is the typedef:

ctypedef void (*cb_type)(void*, shared_ptr[my_cpp_class[specific_type]])

Hopefully this will be useful to someone in a similar boat. Thanks to @DavidW for getting me on the right track.