Trouble importing extension compiled with numpy.distutils

Question

I have a project directory structure:

myproject/
  setup.py
  myproject/
     editors/
       ....
     utilities/
       ...
       find_inf.f90

All the files in the project are python, except for the one fortran file that i have indicated. Now, I can use setuptools to install my project without the fortran file just fine, but to include the fortran file i have to use numpy.distutils.core.Extension. So I have a setup files like this:

from setuptools import find_packages
from numpy.distutils.core import Extension

ext1 = Extension(name = 'myproject.find_inf',
                 sources = ['myproject/utilities/find_inf.f90'], 
                 extra_f90_compile_args=['-fopenmp',],
                 libraries=['gomp'])

if __name__ == "__main__":
    from numpy.distutils.core import setup
    setup(name = 'myproject',
          packages=find_packages(),
          package_data={
              ......
          },
          entry_points={
              'console_scripts': [....]
          },

          ext_modules = [ext1]
          )

This creates and installs myproject-2.0-py2.7-macosx-10.6-x86_64.egg under the site-packages directory and the directory structure looks like:

myproject-2.0-py2.7-macosx-10.6-x86_64.egg
      myproject
          editors\  
          find_inf.pyc
          find_inf.so.dSYM/  
          find_inf.py   
          find_inf.so*
          __init__.py  
          __init__.pyc

So it looks to me that I should be able to import find_inf from myproject. But i can't! Writing from myproject import find_inf produces an import error. What am I doing wrong?

UPDATE:

If I chance the name of the extension from my project.find_inf to just find_inf then the installation puts the extension directly in myproject-2.0-py2.7-macosx-10.6-x86_64.egg. If then I manually move the find_inf files from there to myproject-2.0-py2.7-macosx-10.6-x86_64.egg/myproject then I can import the extension. I still can't make sense of this. Something is clearly wrong in my setup.py that it is not putting the extension in the right place....

UPDATE: Figured it out. Answer below.

The traceback is quite terse, it's just: ``ImportError Traceback (most recent call last) in () ----> 1 from myproject import find_inf ImportError: cannot import name find_inf`` — deepak, Jan 03 '16 at 01:59
Are sure `find_inf.f90` is really in `myproject` and not in a sub directory? — Mike Müller, Jan 03 '16 at 09:07
No, the relative source location is ``myproject/utilities/find_inf.f90`` as is used in the constructor to ``Extension``. — deepak, Jan 03 '16 at 15:44

score 0 · Answer 1 · edited May 23 '17 at 12:15

Okay, figured it out. The name= argument to Extension should have been name=myproject.utilities.find_inf.

Reason: So the issue is that there is no package named myproject.find_inf that setup() is aware of. The packages= argument to setup gets the names of the packages (as a list) and myproject.find_inf wasn't in the list because there is no directory under myproject called find_inf (as the directory structure in my question shows). This answer sheds important light on this issue. In order to have the compiled extensions placed under an appropriate sub-package, one needs:

those sub-packages to be present in the source directory structure
__init__.py files should exist in those packages, and
the names of those package have to be passed to the packages= argument in setup().

i.e. simply describing the extension hierarchy in the call to Extension() is not enough.

Trouble importing extension compiled with numpy.distutils

1 Answers1