3

The Cython project I am currently working on includes some 20 C++ files that act as C++ implementation. On top I have built three Extension Modules written in Cython, each of them capturing different but interconnected functionalities of the C++ implementation. In order to wrap the required functionalities form C++ I used one single .pxd file that defines all the cdef extern blocks. Later on, the 3 .pyx files with the 3 Extension Modules (one Extension Module per .pyx file) cimport the pxd wrapper and add extra functionality.

Now, my objective is to pack the whole library and distribute it using PyPi. My problem arises when using setup.py: I don't know how to make the C++ code available for all Extension Modules. For development I have installed the C++ as a dynamic library in Linux, which clearly has a lot of portability issues. I am more than willing to export the C++ source code to be compiled locally (thus creating a source distribution using python setup.py sdist).

These are the approaches I've taken so far, all of them failed:

  1. Including the C++ source files in the sources of all extensions. It compiles all the C++ files for each Extension. This creates multiple copies of global objects.
  2. Including the C++ source files in the most important Extension and importing it always first. The other Extension modules throw undefined symbol errors because they don't have access to the C++ implementation.
  3. Creating a 4th Extension module with the name of the Cython .pxd wrapper and importing it first. Same result as above (#2).

Any tips will be highly appreciated!

ibarrond
  • 6,617
  • 4
  • 26
  • 45
  • 1
    Tests 2 and 3 sound fine on the principle. There are many projects on PyPI that distribute C or C++ code that is built on the destination machine. For 2 and 3 however, you must not access the C++ code as "cdef extern" but by "cimporting" it as it is already available at the Cython level. Maybe that was the issue? For more help, we'd need a [Minimal, Complete, and Verifiable example](https://stackoverflow.com/help/mcve) – Pierre de Buyl Aug 02 '18 at 08:43
  • Yeap, I'm accessing the C++ code by "cimporting". The only "cdef extern" blocks are located in the `.pxd` wrapper. Could you redirect me to any of those projects? I believe that a working example could help me gratly. So far I've tried checking on lxml, sciPy, spaCy and some others. I'll prepare an example otherwise. – ibarrond Aug 02 '18 at 08:51
  • For the specific case of accessing global variables, it is a bit harder to find examples actually. The simulation code [ESPResSo](https://github.com/espressomd/espresso) uses global variables, see `src/python/espressomd/globals.pxd` there. They use a more complex build procedure but for a simple C++ project that should not be an issue. – Pierre de Buyl Aug 02 '18 at 09:37
  • That problem is not uncommon when using cython. However, it seems as if a good answer is lacking here on SO. Probably adding a [mcve] would help to understand the problem better (the minimal-part is quite important though). – ead Aug 03 '18 at 07:51
  • A somewhat related answer https://stackoverflow.com/a/45655654/5769463, however it isn't concise enough too be of much use. – ead Aug 03 '18 at 07:54
  • If your pxd-file has `cdef extern from header.h` using the functionality will result in going directly to C++-functionality and thus *.so-library is needed during linkage/run time. If you wrap the functionality in cdef-functions and expose only those wrappers in the pxd-file, you will not need the so-library directly it would be enough to cimport the wrapper-extension. However, not sure this could work for something more complex (classes) than simple functions. – ead Aug 03 '18 at 08:03
  • 1
    When I face this issue, I just put all extensions into one to avoid the hassle... With time I got pretty good at muting the voice of the purist in my head, which screams every time I do it. – ead Aug 03 '18 at 08:07
  • @ead How do you do that? Do you have any open source code I could have a look into? That was my first intention but I run into trouble and found everywhere people telling me to use one Extension per `.pyx` file! – ibarrond Aug 09 '18 at 09:54
  • I put them all into the same pyx-file. This is the part where the voice start screaming:) – ead Aug 09 '18 at 10:53
  • Ugh, if I do that with 4000+ lines of code, the next developer touching my code is going to kill me – ibarrond Aug 09 '18 at 10:54
  • @ead I'm back with this old problem. Do you have any particular answers in SO that could point me in the right direction? I had a look at https://stackoverflow.com/questions/63875206/how-to-share-a-c-singleton-between-multiple-c-extensions but couldn't find a solution. I can prepare an example for my case otherwise – ibarrond Sep 30 '20 at 15:40
  • 1
    The approach from the linked answer is well suited for libraries with a very small public interface and works best with C rather than C++ (classes being kind of a problem). So the options I can thing of are either to distribute precompiled so/dll (which reduces the number of platforms for which the extension can be used) or to override `build_clib` to build shared libraries. My plan was to add a second answer, which describes `build_clib`-option in more detail, but haven't found the time yet. Here is a prototype for such solution: https://github.com/realead/commonso/blob/master/setup.py – ead Sep 30 '20 at 19:25
  • @ead, I just found the Holy Grial you were hiding in your garden https://github.com/realead/solinks . Have you ever thought of writing a book about Cython? – ibarrond Oct 01 '20 at 16:33
  • One small extra question: I tried your approach from github.com/realead/commonso/blob/master/setup.py , and while it works perfectly in Linux (Ubuntu), it fails for Windows, complaining that it cannot find the `*.lib` file. Indeed, when compiling the shared `dll` library in windows, I only get a `dll` file, and not a `lib` file. You won't happen to know how to generate both the `dll` and the `lib` files, do you? – ibarrond Oct 01 '20 at 20:24
  • Note: Since this is getting extremely complex, once I get a working solution I will prepare a smaller example and post it at SO for further reference. – ibarrond Oct 01 '20 at 20:25
  • INFO: I ended up building it as a static library (much like in https://stackoverflow.com/questions/57673283/prevent-double-compilation-of-c-files-in-cython). Dynamic libraries need to be stored and found on all OS. I couldn't find a clean way to do this. – ibarrond Oct 02 '20 at 19:51

0 Answers0