2

I am writing a wrapper over c libriary and this lib has file with almost all functions, let say, all_funcs.c. This file in turn requires compilation of lots of another c files

I have created all_funcs.pyx, where I wraped all functions, but I also want to create a submodule, that has access to functions from all_funcs.c. What works for now is adding all c-files to both Extensions in setup.py, however each c-file compiles twice: first for all_funcs.pyx and second for submodule extension.

Are there any ways to provide common sourse files to each Extension?

Example of current setup.py:

ext_helpers = Extension(name=SRC_DIR + '.wrapper.utils.helpers',
                        sources=[SRC_DIR + '/wrapper/utils/helpers.pyx'] + source_files_paths,
                        include_dirs=[SRC_DIR + '/include/'])


ext_all_funcs = Extension(name=SRC_DIR + '.wrapper.all_funcs',
                          sources=[SRC_DIR + '/wrapper/all_funcs.pyx'] + source_files_paths,
                          include_dirs=[SRC_DIR + '/include/'])

EXTENSIONS = [
    ext_helpers,
    ext_all_funcs,
]

if __name__ == "__main__":
    setup(
        packages=PACKAGES,
        zip_safe=False,
        name='some_name',
        ext_modules=cythonize(EXTENSIONS, language_level=3)
        )

source_files_paths - the list with common c source files

Ivan Mishalkin
  • 1,049
  • 9
  • 25

1 Answers1

2

Note: this answer only explains how to avoid multiple compilation of c/cpp-files using libraries-argument of setup-function. It doesn't however explain how to avoid possible problems due to ODR-violation - for that see this SO-post.

Adding libraries-argument to setup will trigger build_clib prior to building of ext_modules (when running setup.py build or setup.py install commands), the resulting static library will also be automatically passed to the linker, when extensions are linked.

For your setup.py, this means:

from setuptools import setup, find_packages, Extension
...
#common c files compiled to a static library:
mylib = ('mylib', {'sources': source_files_paths}) # possible further settings

# no common c-files (taken care of in mylib):
ext_helpers = Extension(name=SRC_DIR + '.wrapper.utils.helpers',
                        sources=[SRC_DIR + '/wrapper/utils/helpers.pyx'],
                        include_dirs=[SRC_DIR + '/include/'])

# no common c-files (taken care of in mylib):
ext_all_funcs = Extension(name=SRC_DIR + '.wrapper.all_funcs',
                          sources=[SRC_DIR + '/wrapper/all_funcs.pyx'],
                          include_dirs=[SRC_DIR + '/include/'])

EXTENSIONS = [
    ext_helpers,
    ext_all_funcs,
]

if __name__ == "__main__":
    setup(
        packages=find_packages(where=SRC_DIR),
        zip_safe=False,
        name='some_name',
        ext_modules=cythonize(EXTENSIONS, language_level=3),
        # will be build as static libraries and automatically passed to linker:
        libraries = [mylib] 
        )

To build the extensions inplace one should invoke:

python setupy.py build_clib build_ext --inplace

as build_ext alone is not enough: we need the static libraries to build before they can be used in extensions.

ead
  • 32,758
  • 6
  • 90
  • 153