Given is a (set of) Python3 packages that is to be deployed in different scenarios either cythonized or as original scripts; the source are pure Python3 sources. Preferably, I would like to use the same setup.py, if possible.
| use case | in-place | include .py modules | cythonized .so modules | 
|---|---|---|---|
1. development pip3 install -e . | 
yes | yes | |
2. "unoptimized" install pip3 install . | 
yes | ||
3. cythonized install pip3 install . --install-option="cythonize" | 
no (except __init__.py) | 
yes | |
4. build (binary) wheel python3 setup.py bdist_wheel | 
no (except __init__.py) | 
yes | 
So far, I succeeded in building a binary distribution wheel with only the cythonized .so shared library and without the original .py module files, following Package only binary compiled .so files of a python library compiled with Cython
. This covers use case #4 and is handled by class build_py.
However, I would also cover #1, #2 and maybe #3; #3 might be better tackled by separately building the bdist_wheel and then installing this, of not otherwise possible in a single step.
# https://stackoverflow.com/a/56043918
from setuptools.command.build_py import build_py as build_py_orig
try:
    from Cython.Build import cythonize
except:
    cythonize = None
from setuptools.command.install import install as install_orig
# https://stackoverflow.com/a/56043918
extensions = [
    Extension('spam.*', ['spam/**/*.py'],
              extra_compile_args=["-O3", "-Wall"]),
]
cython_excludes = ['spam/**/__init__.py']
def not_cythonized(tup):
    (package, module, filepath) = tup
    return any(
        fnmatch.fnmatchcase(filepath, pat=pattern) for pattern in cython_excludes
    ) or not any(
        fnmatch.fnmatchcase(filepath, pat=pattern)
        for ext in extensions
        for pattern in ext.sources
    )
class build_py(build_py_orig):
    def find_modules(self):
        modules = super().find_modules()
        return list(filter(not_cythonized, modules))
    def find_package_modules(self, package, package_dir):
        modules = super().find_package_modules(package, package_dir)
        return list(filter(not_cythonized, modules))
class install(install_orig):
    def finalize_options(self):
        super().finalize_options()
        self.distribution.ext_modules = None
setup(
    name='spam',
    packages=find_packages(),
    ext_modules=cythonize(
        extensions,
        exclude=cython_excludes,
        compiler_directives={
            "language_level": 3,
            "always_allow_keywords": True,
        },
        build_dir="build",  # needs to be explicitly set, otherwise pollutes package sources
    ) if cythonize is not None else [],
    cmdclass={
        'build_py': build_py,
        'install': install,
    },
    include_package_data=True,
    install_requires=[...]
)
The problems I'm facing here:
for use cases #1 and #2 I don't want to cythonize, so
ext_modules=should not be specified/set.What is a sensible way to handle
ext_modules=in this situation? I find it hard to detect the requested operation (install,install -e,develop) before callingsetup(), so would it be better to inherit and override theinstallanddevelopclasses?If the latter, is it possible and allowed to clear the
ext_modulesand how do I avoid prematurely evaluatingcythonize(...)?
in use case #2 with the above code pip3 decides to build an egg which unfortunately includes the
.so's. Might this be due tocythonize(...)getting evaluated in any case? can I avoid building the egg or how do I prevent the egg build process from including the shared libs?this currently includes both the sources (which I don't want to be include) as well as the cythonized modules: how can I prevent the
installclass from installing most of the source modules, yet installing the__init__.pys?