Compiling an optional cython extension only when possible in setup.py

Question

I have a python module fully implemented in python. (For portability reasons.)

The implementation of a small part has been duplicated in a cython module. To improve perfomance where possible.

I know how to install the .c modules created by cython with distutils. However if a machine has no compiler installed, I suspect the setup will fail even though the module is still usable in pure python mode.

Is there a way to compile the .c module if possible but fail gracefully and install without it if compiling is not possible?

score 9 · Accepted Answer · edited Dec 07 '19 at 06:26

I guess you will have to make some modification both in your setup.py and in one __init__ file in your module.

Let say the name of your package will be "module" and you have a functionality, sub for which you have pure python code in the sub subfolder and the equivalent C code in c_sub subfolder. For example in your setup.py :

import logging
from setuptools.extension import Extension
from setuptools.command.build_ext import build_ext
from distutils.errors import CCompilerError, DistutilsExecError, DistutilsPlatformError

logging.basicConfig()
log = logging.getLogger(__file__)

ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError, IOError, SystemExit)

setup_args = {'name': 'module', 'license': 'BSD', 'author': 'xxx',
    'packages': ['module', 'module.sub', 'module.c_sub'],
    'cmdclass': {'build_ext': build_ext}
    }

ext_modules = [Extension("module.c_sub._sub", ["module/c_sub/_sub.c"])]

try:
    # try building with c code :
    setup(ext_modules=ext_modules, **setup_args)
except ext_errors as ex:
    log.warn(ex)
    log.warn("The C extension could not be compiled")

    ## Retry to install the module without C extensions :
    # Remove any previously defined build_ext command class.
    if 'build_ext' in setup_args['cmdclass']:
        del setup_args['cmdclass']['build_ext']

    # If this new 'setup' call don't fail, the module 
    # will be successfully installed, without the C extension :
    setup(**setup_args)
    log.info("Plain-Python installation succeeded.")

Now you will need to include something like this in your __init__.py file (or at any place relevant in your case):

try:
    from .c_sub import *
except ImportError:
    from .sub import *

In this way the C version will be used if it was build, other-wise the plain python version is used. It assumes that sub and c_sub will provide the same API.

You can find an example of setup file doing this way in the Shapely package. Actually most of the code I posted was copied (the construct_build_ext function) or adapted (lines after) from this file.

For my case I could remove the two if-cases in the last exception block, but I had to instead do `del setup_args['ext_modules']`. Could you explain what your code is doing? — LSchueler, Oct 30 '18 at 12:17

MrPisarik · Answer 2 · 2019-12-07T01:25:58.987

3

Class Extension has parameter optional in constructor:

optional - specifies that a build failure in the extension should not abort the build process, but simply skip the extension.

Here is also a link to the quite interesting history of piece of code proposed by mgc.

edited Dec 07 '19 at 01:25

answered Dec 07 '19 at 01:14

MrPisarik

1,260
1
11
21

score 1 · Answer 3 · edited May 23 '17 at 11:45

1

The question How should I structure a Python package that contains Cython code

is related, there the question is how to fallback from Cython to the "already generated C code". You could use a similar strategy to select which of the .py or the .pyx code to install.

edited May 23 '17 at 11:45

Community

1
1

answered Jan 21 '17 at 11:57

Pierre de Buyl

7,074
2
16
22

I do not see how this is applicable. They try importing cython as a python module and fall back to the C-module if importing fails. How do you suggest I try importing the system C compiler in python? – ARF Jan 21 '17 at 14:35
Indeed, I jumped to conclusions a bit early. – Pierre de Buyl Jan 21 '17 at 16:19

Compiling an optional cython extension only when possible in setup.py

3 Answers3

Linked