2

I have inherited a complex setup.py which uses various python modules inside a separate folder called buildchain. The entire repository is basically structured like

- setup.py
- buildchain
   - __init__.py
   - extension_generator.py
- src
   - my_package
      - __init__.py
      - my_module.py

with setup.py stating

import setuptools
from buildchain.extension_generator import CppExtensionGenerator
setuptools.setup(
    ...
    ext_modules=CppExtensionGenerator().create_ext_modules()
)

This structure works fine when running python setup.py build_ext and similar things, but when installing the package using pip, for example in a tox environment, I get a ModuleNotFoundError: No module named 'buildchain'. Understandable, but frustrating.

buildchain/ is explicitly excluded in the manifest, because it is only necessary for compiling the extensions, but not otherwise needed after the installation is finished, so I don't want it in the wheel. The working solution is sticking all code inside setup.py, but that's not easy to navigate. I'm looking for something with the equivalent effect of still not adding all this code to the users's Python environment permanently.

How can I have a setup.py split into different modules with imports between them?

Anaphory
  • 6,045
  • 4
  • 37
  • 68
  • It's unclear why you get the error in the first place. Is buildchain/ not added to the list of files (manifest or the like), that tox can't find it? – 9769953 Oct 24 '22 at 11:50
  • `buildchain/` is explicitly excluded in the manifest, because it is only necessary for compiling the extensions, but not otherwise needed after the installation is finished. The working solution is sticking all code inside `setup.py`, but that's not easy to navigate. I'm looking for something with the equivalent effect of still not adding all this code to the users's Python environment permanently. – Anaphory Oct 24 '22 at 11:55
  • "still not adding all this code to the users's Python environment permanently": do you mean the user environment (where the installed package ends up)? Or some other environment? – 9769953 Oct 24 '22 at 12:01
  • Yes, I mean the user environment. – Anaphory Oct 24 '22 at 12:20
  • But then I fail to understand the problem: the whole buildchain directory would only be downloaded/used to create the extension; it won't be installed. There is a difference between what is in the manifest file, and what setup.py installs. The whole project directory would be downloaded/copied into a temporary directory (when using tox, pip install ... or similar), then everything build, installed, and then the temp directory cleaned up again. So I think you can safely and easily include the buildchain directory with its files into the manifest file. – 9769953 Oct 24 '22 at 12:31
  • `buildchain` should be in the `sdist` but not in the `wheel`. If you get this correctly, it might work (or at least help). – sinoroc Oct 24 '22 at 14:09
  • Okay, that makes some sense to me! I'll try to figure out whether I can get it that way. – Anaphory Oct 24 '22 at 15:01

1 Answers1

0

@sinoroc and @9769953 pointed out that I do not want buildchain/ inside the wheel, but I do want it inside the sdist. Therefore it should be included in the manifest (which informs sdist), but not constitute package data.

How to achieve that is a question answered eg. in Include files in sdist but not wheel.

[I think. I intend to try it out and then amend this answer describing what exactly I needed to do.]

Anaphory
  • 6,045
  • 4
  • 37
  • 68
  • https://stackoverflow.com/a/54953494 and https://stackoverflow.com/a/58050701 also https://sinoroc.gitlab.io/kb/python/package_data.html – sinoroc Oct 27 '22 at 08:14