4

I am working on a python project which contains two modules. The modules are very closely related and hence I want them to be in the same git repo, and to be able to develop them together in my IDE:

  • module1 depends on module2
  • module1 has lots of other heavy dependencies, module2 does not
  • module1 and module2 will be used in different runtime environments
  • module2 should be installable separately so it can run in e.g. an AWS Lambda

Consequently I have tried to set up a project structure which contains the two modules in two folders within one repo, each folder having a setup.py so it can packaged. Is this a reasonable approach?

\module1
  setup.py
  \module1
    __init__.py
    [scripts].py
\module2
  setup.py
  \module2
    __init__.py
    [scripts].py

The above structure should allow me to work on the project locally treating module2 as a regular module, but the setup.py file means that it can be distributed as it's own package, right?

The problem is that I can't include the module2 dependency in module1's setup.py:

from setuptools import setup

setup(
    name="module1",
    version="0.1",
    packages=['module1'], # I really need to include module2 scripts here, right?...
    install_requires=['pandas', 'numpy', ...]
)

Does anyone have any advice on how to approach this problem? The two solutions I can guess are:

  • Packaging and publishing module2 on it's own before depending on it from module1. This makes development much more infleixble
  • Nesting module2 within module1 so I can include it in the packages argument to the setup(...) function. This breaks the clarity that module1 should be treating module2 as if it were essentially an external dependency...
Matthias
  • 4,481
  • 12
  • 45
  • 84
rjmurt
  • 1,135
  • 2
  • 9
  • 25

1 Answers1

4

Your main reason to keep them in the same repository is to enable parallel work on both the modules in your IDE. I would suggest to keep the modules in separate repositories and use the editable mode for pip or develop mode for setuptools to achieve the "parallel" development.

Essentially, you would be installing module2 in a develop/editable mode when you use it in module1. Thereafter, you will see all the changes made to module2 immediately while developing module1

Doing the same will also make sure that things like pip install git+<git_directory> do not break and you can install directly from git

Rubbal
  • 779
  • 7
  • 19
  • Thanks @Rubbal. So if I was to use two separate repos and setuptools develop mode, how would I declare `module1`'s dependency on `module2` in `setup.py`? I understand using `pip -e [path]`, but what's the equivalent in `setup.py`? – rjmurt May 17 '17 at 10:08
  • You will simply add it to `install_requires` (I used to keep a requirements.txt so that I can directly install all requirements while developing. In setup.py, I would parse the requirements.txt and add those to install_requires) – Rubbal May 17 '17 at 10:13
  • I think this is probably the right answer (although I never got `install_requires` working properly). I guess my question is really whether there is/should be a mechanism for building more than one PyPI style package from a single set of code. It seems like using setuptools for the build step isn't that flexible... – rjmurt May 18 '17 at 09:43