2

I have written 3 Cython extensions, and I would like to package them in a tidy way that I can distribute to coworkers to use in Python, without them needing a C compiler installed.

Directory structure is:

myproject\
|---- setup.py
|---- gamma-source\
|      |---- eta.pxd
|      |---- eta.pyx
|      |---- omega.pxd
|      |---- omega.pyx
|      |---- rho.pxd
|      |---- rho.pyx

Where omega has a cimport from rho, eta has a cimport and an import from omega, rho has cimports from eta and omega.

My setup.py is:

from setuptools import setup, Extension
from Cython.Build import cythonize

__version__ = '0.1.0'

exts = [Extension("eta",
                  sources = ["gamma-source\\eta.pyx"]),
        Extension("omega",
                  sources = ["gamma-source\\omega.pyx"]),
        Extension("rho",
                  sources = ["gamma-source\\rho.pyx"]),
       ]

setup(name="gamma",
      version=__version__,
      ext_modules = cythonize(exts,
                              compiler_directives = {'language_level':'3'}
                              ),
      zip_safe = False,
      )

and using python setup.py install gets me working on my own machine, but:

  1. The .pyd files are dropped into my site-packages folder rather than being in site-packages\rho, which is untidy.
  2. This method requires me to import eta, omega, rho instead of from gamma import omega or import gamma.omega, which I would prefer if possible.

I tried changing setup.py to:

exts = [Extension("gamma.eta",
                  sources = ["gamma-source\\eta.pyx"]),
        Extension("gamma.omega",
                  sources = ["gamma-source\\omega.pyx"]),
        Extension("gamma.rho",
                  sources = ["gamma-source\\rho.pyx"]),
       ]

but python setup.py develop gives me:

error: could not create 'gamma\rho.cp39-win_amd64.pyd': No such file or directory.

I then tried putting a gamma directory into myproject, and then it would install, but I found that from gamma import omega worked but importing either eta and rho gave me

ModuleNotFoundError: No module named 'rho'

I then tried changing the imports from from omega cimport omega_class to from .omega cimport omega_class in the hope that that would get the modules to find each other, but that wouldn't compile - eta.pyx couldn't find omega_class.

I tried adding a blank __init__.py file to gamma-source, but that also wouldn't compile.

Where am I going wrong? How should I package these extensions into a coherent unit? I'm happy to share more detailed error info from any of the approaches I took, but not sure which of them would be fruitful.

This question looked like it might be hitting the same issue, but it wasn't clear to me how I would apply the given solution: Building Python package containing multiple Cython extensions

Jeremy Sadler
  • 55
  • 4
  • 1
    Instead of gamma_source, try renaming the package gamma. Add an __init__.py so that cython knows it is a package, Also, do relative cimports (from .omega cimport omega_class). What is the error in this case? – Golden Rockefeller Apr 28 '21 at 16:26
  • https://github.com/jakevdp/cython_template worked for me in a somewhat similar case – ev-br Apr 29 '21 at 05:17
  • @Golden Rockefeller Renamed the package `gamma`. Added a blank `__init__.py`. Using the relative cimports gave me "relative cimport beyond main package is not allowed." Changing the cimports to "from gamma.omega cimport omega_class" successfully built, but at run time I got "ImportError: cannot import name 'rho; from 'gamma' ()" Changing the extension names to `gamma.omega` etc successfully built, but at run time I got "ValueError: gamma.omega.omega_class size changed, may indicate binary incompatibility. Expected 56 from C header, got 24 from PyObject". – Jeremy Sadler Apr 29 '21 at 09:11
  • @GoldenRockefeller Aha! Having the extension names as `gamma.omega` *and* the relative imports *and* the blank `__init__.py` worked! Thank you - I was banging my head on this for 2 days, going round and round in circles. – Jeremy Sadler Apr 29 '21 at 09:14

0 Answers0