10

I want to create a package for a project that does not contain any .py source files, but is completely implemented as a Python C extension (resulting in an .so). Additionally, assume that the .so is already built by a separate build process (say CMake).

I know that setuptools/distutils minimally requires a directory structure:

  • mymodule
    • __init__.py

But what I really want is for mymodule to be provided by a C extension (say mymodule.so) such that after installing the package, import mymodule has the same effect as directly importing mymodule.so.

I know that I could have this kind of directory structure:

  • mymodule
    • __init__.py
    • mymodule_native.so

and have __init__.py be:

from mymodule_native import *

This kind of works, but an object A imported from mymodule will actually look like mymodule.mymodule_native.A.

Is there a more direct way?

jogloran
  • 952
  • 1
  • 12
  • 23
  • What setuptools/distutils functionality are you hoping to take advantage of? – user2357112 Aug 09 '17 at 21:55
  • 1) Being able to install through pip with the same workflow as other packages, and 2) being able to package pure Python code together with the C extension. Of course I can just copy the `.so` onto the PYTHONPATH but I was hoping that distutils could handle the path munging. – jogloran Aug 09 '17 at 21:57
  • 1
    Are you _sure_ that the `import *` way doesn't work? I just tried it and `mymodule.A` and `mymodule.mymodule_native.A` point to same exact object. – Dmiters Aug 15 '17 at 00:05
  • They do point to the same object -- I was wondering if there's a preferred way apart from this, which seems like a workaround. Ideally, it would be indistinguishable from being directly in the package `mymodule`. – jogloran Aug 15 '17 at 00:06
  • Ooh, try compiling it with the name `__init__` so you get `__init__.so` (and so it has an `init__init__` function inside). Then you can place it in your module directory, and create a blank `__init__.py` just so it gets recognized as a module. – Dmiters Aug 15 '17 at 00:16
  • Why do you think that setuptools requires a directory? – Manuel Jacob Aug 17 '17 at 20:11
  • It does require a directory with a module structure (`__init__.py`): otherwise you get `warning: install_lib: 'build/lib.linux-x86_64-2.7' does not exist -- no Python modules to install` – jogloran Aug 17 '17 at 23:55

1 Answers1

1

It's possible if the extension is configured by setuptools.

For example:

from setuptools import setup, Extension

extension = Extension('mymodule', sources=[<..>])
setup('mymodule', ext_modules=[extension])

Once installed, the extension is available as import mymodule. Note that find_packages is not used.

This needs to be done by setuptools as it would otherwise require a packages setting if no ext_modules are provided.

However, this makes the .so module be installed directly under site-packages directory and will conflict with any non-extension python modules of the same name.

This is generally considered bad practice and most libraries use a bare python module with a single __init__.py, under which the extension is available.

You may in future add python code to your module for example and wish to separate the pure python code from extension code. Or you may want to add more than one extension which is not possible in this way, at least not while keeping the same package name.

So a structure like mymodule.<python modules> and mymodule.my_extension makes sense.

Personally I'd have separate name spaces for extension code vs python code and not do from <ext mod> import * in __init__.py.

danny
  • 5,140
  • 1
  • 19
  • 31
  • 2
    Hi there, thanks for your answer. I was more interested in doing this when the .so is not built by `Extension` -- I suppose the only way is to copy it into the module directory then access it as `mymodule.mymodule_native`. – jogloran Aug 21 '17 at 21:06