How can I manually compile Cython code that uses C++?

Question

I have exactly copied the example code given in the Cython documentation for wrapping C++ classes. I can successfully build and import the rect.so extension using distutils and the cythonize() method, i.e. by:

1. Putting the following directives at the top of rect.pyx:

   # distutils: language = c++
   # distutils: sources = Rectangle.cpp
   

2. Writing a setup.py file that contains this:

   from distutils.core import setup
   from Cython.Build import cythonize
   
   setup(
       name = "rectangleapp",
       ext_modules = cythonize('*.pyx'),
   )
   

3. Calling

   $ python setup.py build_ext --inplace
   

However, when I'm wrapping C code in Cython I often find it more convenient to compile individual extensions manually from the command line, i.e.:

1. Generate the .c code using the command line Cython compiler

   $ cython foo.pyx
   

2. Manually compile it using gcc:

   $ gcc -shared -fPIC -O3 -I /usr/lib/python2.7 -L /usr/lib/python2.7 \
          foo.c -lpython2.7 -o foo.so
   

I've tried applying the same process to build the rect.so example above:

$ cython --cplus rect.pyx
$ g++ -shared -fPIC -O3 -I /usr/lib/python2.7 -L /usr/lib/python2.7 \
      rect.cpp -lpython2.7 -o rect.so

Both the Cython and g++ compilation steps seem to succeed - I don't get any command line output, and at the end I have a rect.so built. However, when I then try to import the module I get an undefined symbol error:

In [1]: import rect
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-ba16f97c2145> in <module>()
----> 1 import rect

ImportError: ./rect.so: undefined symbol: _ZN6shapes9Rectangle9getLengthEv

What's the correct procedure for manually compiling Cython code that wraps C++ classes?

Answer 1

The problem here is that you said that somewhere you will provide the definition of a class called Rectangle -- where the example code states

cdef extern from "Rectangle.h" namespace "shapes":
    cdef cppclass Rectangle:
        ...

However, when you compiled the library you didn't provide the code for Rectangle, or a library that contained it, so rect.so has no idea where to find this Rectangle class.

To run your code you must first create the Rectangle object file.

gcc -c Rectangle.cpp # creates a file called Rectangle.o

Now you can either create a library to dynamically link against, or statically link the object file into rect.so. I'll cover statically linking first as it's simplest.

gcc -shared -fPIC -I /usr/include/python2.7 rect.cpp Rectangle.o -o rect.so

Note that I haven't included the library for python. This is because you expect your library to be loaded by the python interpreter, thus the python libraries will already be loaded by the process when your library is loaded. In addition to providing rect.cpp as a source I also provide Rectangle.o. So lets try running a program using your module.

run.py

import rect
print(rect.PyRectangle(0, 0, 1, 2).getLength())

Unfortunately, this produces another error:

ImportError: /home/user/rectangle/rect.so undefined symbol: _ZTINSt8ios_base7failureE

This is because cython needs the c++ standard library, but python hasn't loaded it. You can fix this by adding the c++ standard library to the required libraries for rect.so

gcc -shared -fPIC -I/usr/include/python2.7 rect.cpp Rectangle.o -lstdc++ \
     -o rect.so

Run run.py again and all should work. However, the code for rect.so is larger than it needs to be, especially if you produce multiple libraries that depend on the same code. You can dynamically link the Rectangle code, by making it a library as well.

gcc -shared -fPIC Rectangle.o -o libRectangle.so
gcc -shared -fPIC -I/usr/include/python2.7 -L. rect.cpp -lRectangle -lstdc++ \
     -o rect.so

We compile the Rectangle code into a shared library in the current directory and provide -L. so gcc knows to look for libraries in the current directory and -lRectangle so gcc knows to look for the Rectangle library. Finally, to be able to run your code you must tell python where the Rectangle library lives. Before running python enter

export LD_LIBRARY_PATH="/home/user/rectangle" # where libRectangle.so lives

You can use a shell script to make sure this is done every time before you run your program, but it makes things messier. Best to just stick with statically linking Rectangle.