2

I am trying to figure out a good way to package and deploy a number of python packages I created. Eventually, I would like to use some package repository for deployment or create some sort of setup script.

The structure of my project is as follows: I have two subprojects A and B that both use tools from another self-created package C. The tools in C are for internal use only and not of bigger interest to a general audience. However, A and B shall be deployed. I want that users can install A and B independently of each other, but I do not need/want to deploy C as standalone.

In the best case, I would like users to be able to install the packages with something along the lines of 

pip install my_project.A

or 

pip install my_project.B

Furthermore, in A, I would like to import C as follows:

import my_project.C

Would I have to package A, B, and C independently and work with install_requires in setuptools.setup? If all projects belong to a large "meta-project", is there a way I can bundle them together weakly while maintaining their general independence? (In my example, C contains "tools". I do not want to deploy a package with such a generic name. Would I then have to deploy it as my_project_tools?)

Is there a way so that all packages installed from that meta-package are installed in the same folder hierarchy while allowing the installation of separate components of that meta-package? There is a related answer here, but my problem differs in that there is the common dependency C.

I am new to packaging, so I appreciate answers that do not suppose too much background knowlege.

Additional info: I am using Python 3.7, and the program is not compatible with lower versions. It should run platform independently, though. Some modules require compilation and Cython.

Samufi
  • 2,465
  • 3
  • 19
  • 43

1 Answers1

2

Something like the following could be a solution, in which Nmspc could be considered the meta-project of Ping and Pong. It's not actually tested, so some tweaking will be needed.

The directory tree might look like this:

.
├── NmspcCore
│   ├── nmspc
│   │   └── _core
│   │       ├── __init__.py
│   │       └── common.py
│   └── setup.py
├── NmspcPing
│   ├── nmspc
│   │   └── ping
│   │       ├── __init__.py
│   │       └── foo.py
│   └── setup.py
└── NmspcPong
    ├── nmspc
    │   └── pong
    │       ├── __init__.py
    │       └── bar.py
    └── setup.py

3 Python projects:

  • NmspcCore provides nmspc._core (the leading underscore denotes a private element) and should not be installed directly
  • NmspcPing provides nmspc.ping and depends on NmspcCore
  • NmspcPong provides nmspc.pong and depends on NmspcCore

They all make use of namespace packages. They can be designed according to the instructions from the Python Packaging User Guide on "Packaging namespace packages, Native namespace packages". There is another example here.

NmspcCore/setup.py:

#!/usr/bin/env python3

import setuptools

setuptools.setup(
    name='NmspcCore',
    version='1.2.3',
    packages=['nmspc._core',],
)

NmspcCore/nmspc/_core/common.py

MARCO = 'polo'

NmspcPing/setup.py (and similarly NmspcPong/setup.py):

#!/usr/bin/env python3

import setuptools

setuptools.setup(
    name='NmspcPing',
    version='1.2.3',
    packages=['nmspc.ping',],
    install_requires=['NmspcCore',],
)

NmspcPing/nmspc/ping/foo.py

import nmspc._core

print(nmspc._core.common.MARCO)

An umbrella project Nmspc could also be considered. Such a project would be empty (no actual code), but would have NmspcPing and NmspcPong as direct dependencies, so that one could install the whole projects ensemble all at once.

sinoroc
  • 18,409
  • 2
  • 39
  • 70
  • Thanks for this clear answer! I am not sure I fully understand the underscore issue, though. Wat is the benefit of making `_core` private? Could you elaborate a little further? – Samufi Dec 29 '19 at 16:34
  • 1
    1. The leading underscore is just a Python convention. This basically means "for internal use only". When one sees the underscore, they should know not to mess with this, they should for example not import it. But remember that it is just a convention, the underscore has no actual effect and the elements are still importable. 2. The phrasing is probably misleading: the project can and will be installed but there is no point in installing it independently since it is meant as a library for internal use. – sinoroc Dec 29 '19 at 16:41
  • 1
    You might also want to consider an _umbrella_ project _Nmspc_ that would be empty (no actual code), but that would have _NmspcPing_ and _NmspcPong_ as direct dependencies, so that one could install the whole project all at once. – sinoroc Dec 29 '19 at 16:58