Link f2py generated *.so file in a python package using setuptools

Question

I wish to deploy a package to PyPi using setuptools. However, the core part of the package is actually written in Fortran, and I am using f2py to wrap it in python. Basically the project's structure looks like this:

my_project

• license.txt
• README.md
• setup.py
• my_project
  • init.py
  • myfunc.py
  • hello.so

The module myfunc.py imports hello.so (import my_project.hello) which can then be used by functions inside myfunc.py. This works perfectly on my machine.

Then I tried standard setuptools installation: sudo python3 setup.py install on my Ubuntu, and it gets installed perfectly. But unfortunately, while importing, it throws ModuleNotFoundError: No module named 'hello'.

Now, from what I understand, on Linux based systems, for python, the shared libraries *.so are stored in /usr/lib/python3/dist-packages/. So I manually copied this hello.so there, and I got a working package! But of course this works only locally. What I would like to do is to tell setuptools to include hello.so inside the python-egg and automatically do the copying etc so that when a user uses pip3 install my_package, they will have access to this shared library automatically. I can see that numpy has somehow achieved that but even after looking at their code, I haven't been able to decode how they did it. Can someone help me with this? Thanks in advance.

Answer 1

You can achieve this with a setup.py file like this (simplified version, keep only the relevant parts for building external modules)

import os
from setuptools import setup, Extension
from setuptools.command.build_ext import build_ext


class f2py_Extension(Extension):

    def __init__(self, name, sourcedirs):
        Extension.__init__(self, name, sources=[])
        self.sourcedirs = [os.path.abspath(sourcedir) for sourcedir in sourcedirs]
        self.dirs = sourcedirs

class f2py_Build(build_ext):

    def run(self):
        for ext in self.extensions:
            self.build_extension(ext)

    def build_extension(self, ext):
        # compile
        for ind,to_compile in enumerate(ext.sourcedirs):
            module_loc = os.path.split(ext.dirs[ind])[0]
            module_name = os.path.split(to_compile)[1].split('.')[0]
            os.system('cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name))

setup(
    name="foo",
    ext_modules=[f2py_Extension('fortran_external',['foo/one.F90','foo/bar/two.F90'])],
    cmdclass=dict(build_ext=f2py_Build),
)

The essential parts for building an external module are ext_modules and cmdclass in setup(...). ext_modules is just a list of Extension instances, each of which describes a set of extension modules. In the setup.py above, I tell ext_modules I want to create two external modules with two source files foo/test.F90 and foo/bar/two.F90. Based on ext_modules, cmdclass is responsible for compiling the two modules, in our case, the command for compiling the module is

'cd %s;f2py -c %s -m %s' % (module_loc,to_compile,module_name)

---

Project structure before installation

├── foo
│   ├── __init__.py
│   ├── bar
│   │   └── two.F90
│   └── one.F90
└── setup.py

Project structure after python setup.py install

├── build
│   └── bdist.linux-x86_64
├── dist
│   └── foo-0.0.0-py3.7-linux-x86_64.egg
├── foo
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-37.pyc
│   ├── bar
│   │   ├── two.F90
│   │   └── two.cpython-37m-x86_64-linux-gnu.so
│   ├── one.F90
│   └── one.cpython-37m-x86_64-linux-gnu.so
├── foo.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
└── setup.py

The two source files one.F90 and two.F90 are very simple

one.F90

module test

  implicit none

  contains

  subroutine add(a)

    implicit none
    integer :: a
    integer :: b
    b = a + 1
    print *, 'one',b

  end subroutine add


end module test

two.F90

module test

  implicit none

  contains

  subroutine add(a)

    implicit none
    integer :: a
    integer :: b
    b = a + 2
    print *, 'two',b

  end subroutine add


end module test

After I installed the package, I can successfully run

>>> from foo.bar.two import test
>>> test.add(5)
 two           7

and

>>> from foo.one import test
>>> test.add(5)
 one           6

Answer 2

Here is an approach based on F2PY's documentation (the example there covers building multiple F2PY modules, and multiple source files per module), making use of numpy.distutils, that supports Fortran source files.

The structure of a minimal example with multiple F2PY extension modules is based on a src directory layout. It is not necessary/required, but has the advantage that the test routine cannot run unless the package has been installed successfully.

Source layout

my_project
|
+-- src
|   |
|   +-- my_project
|       |
|       +-- __init__.py
|       +-- mod1.py
|       +-- funcs_m.f90
|       +-- two
|           |
|           +-- pluss2.f90
|           +-- times2.f90
|
+-- test_my_project.py
+-- setup.py

• setup.py

from setuptools import find_packages

from numpy.distutils.core import setup, Extension

ext1 = Extension(name='my_project.modf90',
                 sources=['src/my_project/funcs_m.f90'],
                 f2py_options=['--quiet'],
                )

ext2 = Extension(name='my_project.oldf90',
                 sources=['src/my_project/two/plus2.f90', 'src/my_project/two/times2.f90'],
                 f2py_options=['--quiet'],
                )

setup(name="my_project",
      version="0.0.1",
      package_dir={"": "src"},
      packages=find_packages(where="src"),
      ext_modules=[ext1, ext2])

• __init__.py

The __init__.py file is empty. (Can e.g. import the F2PY modules here if desired)

• mod1.py

def add(a, b):
  """ add inputs a and b, and return """
  return a + b

• funcs_m.f90

module funcs_m
  implicit none
  contains
    subroutine add(a, b, c)
      integer, intent(in)  :: a
      integer, intent(in)  :: b
      integer, intent(out) :: c
      c = a + b
    end subroutine add
end module funcs_m

• plus2.f90

subroutine plus2(x, y)
  integer, intent(in)   :: x
  integer, intent(out)  :: y
  y = x + 2
end subroutine plus2

• times2.f90

subroutine times2(x, y)
  integer, intent(in)   :: x
  integer, intent(out)  :: y
  y = x * 2
end subroutine times2

• test_my_project.py

import my_project.mod1
import my_project.oldf90
import my_project.modf90

print("mod1.add:            1 + 2 = ", my_project.mod1.add(1, 2))
print("modf90.funcs_m.add:  1 + 2 = ", my_project.modf90.funcs_m.add(1, 2))
x = 1
x = my_project.oldf90.plus2(x)
print("oldf90.plus2:        1 + 2 = ", x)
x = my_project.oldf90.times2(x)
print("oldf90.plus2:        3 * 2 = ", x)

Installing

Now, one can use pip to install the package. There are several advantages to using pip (including ease of upgrading, or uninstalling) as opposed to setup.py install (but this can still be used for building the package for distribution!). From the directory containing setup.py:

> python -m pip install .

Testing

And then, to test the just installed package

> python test_my_project.py
mod1.add:            1 + 2 =  3
modf90.funcs_m.add:  1 + 2 =  3
oldf90.plus2:        1 + 2 =  3
oldf90.plus2:        3 * 2 =  6

This setup has been tested with success on Windows 10 (with ifort), on Ubuntu 18.04 (with gfortran) and on MacOS High Sierra (with gfortran), all with Python 3.6.3.