Python can't import extensions generated in a nested hierarchy

Question

I have a library with a deeply nested module hierarchy, with a structure such as

src/
  my_library/
    __init__.py
    my_module/
      __init__.py
      my_submodule.py

Such that for example one might be able to write from my_library.my_module.my_submodule import Foo etc.

I have a function which walks the file tree, discovering all of python modules and converting them into Extension objects which are passed into cythonize, effectively cythonizing the whole module. Inspecting the before they are built, they look (in dictionary form) something like:

(Pdb) pp vars(e)
{'define_macros': [],
 'depends': [],
 'export_symbols': [],
 'extra_compile_args': [],
 'extra_link_args': [],
 'extra_objects': [],
 'include_dirs': [],
 'language': None,
 'libraries': [],
 'library_dirs': [],
 'name': 'my_library.my_module.my_submodule',
 'optional': None,
 'runtime_library_dirs': [],
 'sources': ['src/my_library/my_module/my_submodule.c'],
 'swig_opts': [],
 'undef_macros': []}

The list of extensions is then passed into the ext_modules argument of setuptools.setup(). Invoking setup.py install results in a bunch of compilations as expected, which is great. However, files are written to a directory structure mirroring their location prior to being compiled, so for example it ends up with something like

lib/python3.4/site-packages/
  my_library.cpython-34m.so
  my_library/
    my_module.cpython-34m.so
    my_module/
      my_submodule.cpython-34m.so

Because of this nested structure, I am only able to import the top-level object when the installed-to directory is in my PYTHONPATH:

>>> import my_library
>>> my_library.foo()
'foo'
>>> import my_library.my_module
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'my_library.my_module'

It seems that since the generated files are written to subdirectories rather than directly to the site-packages folder, the python interpreter is unable to discover them. I don't know much about python extensions, but looking around I didn't see a way to get python to put them in a flat structure rather than nested, nor do I know if there's a way to get python to discover .so files written to subdirectories -- perhaps by adding __init__.py files?

score 2 · Answer 1 · edited Jan 14 '20 at 16:23

2

It looks like you "double define" my_library: once via the __init__.py file and once via the cython code.

The solution is to have a "pure Python" module hierarchy and to create cython modules as elements in that hierarchy, outside any conflicts with the Python modules.

src/   
    my_library/
        __init__.py
        my_first_cython_module.pyx
        my_module/
            __init__.py
            my_submodule.py
            other_cython_module.pyx

This way, you can do import my_library.my_module, etc. For the Cython modules, import my_library.my_first_cython_module, etc.

You can keep your current import names by importing the Cython code from the __init__.py files:

from my_library.my_first_cython_module import foo

edited Jan 14 '20 at 16:23

Roland Puntaier

3,250
30
35

answered Jul 17 '17 at 11:46

Pierre de Buyl

7,074
2
16
22

The thing is that I'm trying to take a pure python library and compile *all* of the code in it with cython. Basically, the goal is to have a library with the same structure as the python one, but built entirely with shared object files. – limp_chimp Jul 17 '17 at 14:21
1

It is impossible, see this other SO post https://stackoverflow.com/questions/30157363/collapse-multiple-submodules-to-one-cython-extension and the Cython FAQ: https://github.com/cython/cython/wiki/FAQ#how-to-compile-cython-with-subpackages – Pierre de Buyl Jul 17 '17 at 14:40

Python can't import extensions generated in a nested hierarchy

1 Answers1