2

Imagine the following folder structure:

/project
  run.py
  /Helper
    Helper.pxd
    Helper.pyx
    setup.py
  /Settings
    Settings.pxd
    Settings.pyx
    setup.py

Helper uses the types Settings and PySettings defined in Settings.pxd. Therefor in Helper.pxd I'm doing:

from Settings cimport Settings, PySettings

In the setup.py in the Helper directory I put

include_path=[".", "../Settings/"]

in the cythonize command. Thus Helper knows about Settings and everything compiles.

In run.py I'd like to import Settings as well as Helper. The Settings import works fine but when doing

import Helper.Helper

I get the following error:

Traceback (most recent call last):
  File "run.py", line 1, in <module>
    import Helper.Helper
  File "../Settings/Settings.pxd", line 6, in init Helper
AttributeError: module 'Settings' has no attribute 'PySettings'

This issue disappears as soon as everything is in the same directory.

For the sake of completeness you'll find the whole code below:

/project/run.py

import Settings.Settings as S
import Helper.Helper as H

settings = S.PySettings()
settings.doSomething()

H.PyMyFunction(settings)

/project/Helper/Helper.pxd

from Settings cimport Settings, PySettings

cdef extern from "../../cppCode/Helper/Helper.h":
  void myFunction(Settings settings)

/project/Helper/Helper.pyx

#distutils: sources = [../../cppCode/Helper/Helper.cpp, ../../cppCode/Settings/Settings.cpp]
#distutils: language = c++

def PyMyFunction(PySettings settings):
  myFunction(settings.c_settings)

/project/Helper/setup.py

from distutils.core import setup
from Cython.Build import cythonize
from setuptools.extension import Extension

extensions = [
  Extension("Helper", ["Helper.pyx"])
]

setup(ext_modules=cythonize(extensions, include_path=[".", "../Settings/"]))

/project/Settings/Settings.pxd

cdef extern from "../../cppCode/Settings/Settings.h":
  cdef cppclass Settings:
    Settings() except +
    void doSomething()

cdef class PySettings:
  cdef Settings c_settings

/project/Settings/Settings.pyx

#distutils: sources = ../../cppCode/Settings/Settings.cpp
#distutils: language = c++

cdef class PySettings:
  def __cinit__(self):
    self.c_settings = Settings()

  def doSomething(self):
    self.c_settings.doSomething()

/project/Settings/setup.py

from distutils.core import setup
from Cython.Build import cythonize
from setuptools.extension import Extension

extensions = [
  Extension("Settings", ["Settings.pyx"])
]

setup(ext_modules=cythonize(extensions))
zyzzler
  • 105
  • 7
  • Did you see [What is __init_.py for?](https://stackoverflow.com/q/448271/5769463) – ead Sep 25 '19 at 07:27
  • 1
    Yes, I also tried to put `__init__.py` files in all the directories. But it didn't make a difference. – zyzzler Sep 25 '19 at 07:54
  • Relevant: https://stackoverflow.com/questions/56115159/pxd-relative-cimport-works-for-language-level-2-but-not-for-3 – 0 _ Jun 30 '21 at 03:37
  • Relevant: https://stackoverflow.com/questions/33555927/cython-relative-cimport-beyond-main-package-is-not-allowed – 0 _ Jun 30 '21 at 03:38
  • Relevant: https://stackoverflow.com/questions/36612483/why-doesnt-my-cython-cimport-for-a-pxd-file-work – 0 _ Jun 30 '21 at 03:38

1 Answers1

6

"include_path" is for C includes. It has no influence on where Python looks for pxd files.

You usually want one setup file, at the top level. A better way of doing it is probably:

setup.py
project/
  run.py
  __init__.pxd
  Settings/
    __init__.pxd
    Settings.pyx/pxd
  Helper/
    __init__.pxd
    Helper.pyx/pxd

This will create a single module called "package" which will all be built at once and installed at once. Once installed your user would be able to do from project import whatever.

You'll notice I've added some __init__.pxd files. These serve basically the same purpose as __init__.py files, except they identify folders as (sub)packages for Cython's cimport mechanism.

Helper .pxd then starts:

from project.Settings.Settings cimport Settings, PySettings

I've used a full rather than relative import because cimport and relative imports seems a bit unreliable.

Setup.py is as follows:

from setuptools.extension import Extension
from setuptools import setup
from Cython.Build import cythonize

ext_modules = cythonize("project/*/*.pyx")

setup(ext_modules=ext_modules)

You are welcome to explicitly use Extension but for something simple it seemed easier to send the file pattern directly to cythonize.

I've only tested the Cythonize stage of this since I don't have the C++ files needed to compile and run it.

0 _
  • 10,524
  • 11
  • 77
  • 109
DavidW
  • 29,336
  • 6
  • 55
  • 86
  • Thank you very much! This works perfectly! I have a few follow up comments: – zyzzler Sep 26 '19 at 11:17
  • 1) run.py (where I use the cython modules in) has to be above the project folder where setup.py is. Otherwise I can not do `from cythonCode.Settings.Settings import PySettings`, right? 2) I don't get it to work when using extensions. I don't know what the problem is. If you don't mind having another look at it I created this git: https://github.com/zyzzler/cython-test.git (feel free to use `source setup_venv.sh` to have all the packages). I have a `setup_with_extensions.py`with my approach so far. 3) I don't understand the output of `run.py`. The constructor is called 3 times. – zyzzler Sep 26 '19 at 11:28
  • 1) I'd imagine that run.py is part of the package, so should be in the project directory (so if you use setup.py to install it should get installed with everything else) - obviously you may need to change the imports in run.py to match where it's located. 2) I think that the names of the extensions should be dotted (e.g. the first argument should be `"project.Settings.Settings"`). 3) I'm not immediately able to test any of this, so I don't know at the moment. – DavidW Sep 26 '19 at 14:39
  • 1) run.py is actually just the file I want to test the modules with. So right now (with run.py in the same location as setup.py) I can do `from cythonCode.Settings.Settings import PySettings` for instance. Putting run.py in the cythonCode folder I don't get the imports to work anymore. But probably I won't need it like this. 2) Thant's it! Works perfectly! Thanks again! – zyzzler Sep 26 '19 at 15:00