69

I'm working toward adopting Python as part of my team's development tool suite. With the other languages/tools we use, we develop many reusable functions and classes that are specific to the work we do. This standardizes the way we do things and saves a lot of wheel re-inventing.

I can't seem to find any examples of how this is usually handled with Python. Right now I have a development folder on a local drive, with multiple project folders below that, and an additional "common" folder containing packages and modules with re-usable classes and functions. These "common" modules are imported by modules within multiple projects.

Development/
    Common/
        Package_a/
        Package_b/
    Project1/
        Package1_1/
        Package1_2/
    Project2/
        Package2_1/
        Package2_2/

In trying to learn how to distribute a Python application, it seems that there is an assumption that all referenced packages are below the top-level project folder, not collateral to it. The thought also occurred to me that perhaps the correct approach is to develop common/framework modules in a separate project, and once tested, deploy those to each developer's environment by installing to the site-packages folder. However, that also raises questions re distribution.

Can anyone shed light on this, or point me to a resource that discusses this issue?

vaultah
  • 44,105
  • 12
  • 114
  • 143
Steve Sawyer
  • 1,785
  • 4
  • 18
  • 21
  • 1
    How do you want the packages presented to the python program? Are Package_a -- Package2_1 all top level packages, or are you trying to do some namespacing / hierarchy? – Matt Anderson Jun 18 '13 at 17:24
  • Hmmm... not sure what you mean by "top level" packages. In the case of the packages under "Common", they're a way of distinguishing classes and functions that are standard library extensions and/or wrappers and decorators of standard library classes, and those classes and functions that are "enterprise" specific, i.e. that provide re-usable functionality that is specific to our environment. Each package has multiple modules within it that further organize the reusable routines within those namespaces. Not sure if I'm answering the question. – Steve Sawyer Jun 19 '13 at 02:11

4 Answers4

23

If you have common code that you want to share across multiple projects, it may be worth thinking about storing this code in a physically separate project, which is then imported as a dependency into your other projects. This is easily achieved if you host your common code project in github or bitbucket, where you can use pip to install it in any other project. This approach not only helps you to easily share common code across multiple projects, but it also helps protect you from inadvertently creating bad dependencies (i.e. those directed from your common code to your non common code).

The link below provides a good introduction to using pip and virtualenv to manage dependencies, definitely worth a read if you and your team are fairly new to working with python as this is a very common toolchain used for just this kind of problem:

http://dabapps.com/blog/introduction-to-pip-and-virtualenv-python/

And the link below shows you how to pull in dependencies from github using pip:

How to use Python Pip install software, to pull packages from Github?

Community
  • 1
  • 1
robjohncox
  • 3,639
  • 3
  • 25
  • 51
  • Thanks - I needed to look into a source/version control system, and this could kill two birds... – Steve Sawyer Jun 19 '13 at 01:58
  • No problem - even if you don't need to go for source control, virtualenv and pip come _highly_ recommended. Good luck – robjohncox Jun 19 '13 at 07:53
  • >>imported as a dependency into your other projects<< Actually, that's what I'm doing currently and it works fine within the IDE, but distributing an application that uses the common code is what has me flummoxed. – Steve Sawyer Jun 19 '13 at 15:20
  • The way we get around this is to store the dependencies in a file at the root of each project called requirements.txt, which lists all the dependencies, this gets distributed, and whenever we deploy the project somewhere, we run **pip install -r requirements.txt** to pull in the dependencies. I think this is a fairly standard approach – robjohncox Jun 19 '13 at 15:45
  • Yup! This kills two birds. A much better design I was desperately looking for. – user2290820 Apr 07 '15 at 14:38
  • I have packaged my shared/common/utils as a separate project and hosted on gitlab, I can even install it as a dependency within another project, however I can't figure out how to setup the import statements in the code so that the other project can call the shared/common/utils and parts of the shared/common/utils can call itself. Is there a simple working example of this somewhere? – MattG Jul 14 '21 at 13:05
11

The must-read-first on this kind of stuff is here:

What is the best project structure for a Python application?

in case you haven't seen it (and follow the link in the second answer).

The key is that each major package be importable as if "." was the top level directory, which means that it will also work correctly when installed in a site-packages. What this implies is that major packages should all be flat within the top directory, as in:

myproject-0.1/
    myproject/
        framework/
    packageA/
        sub_package_in_A/
            module.py
    packageB/
        ...

Then both you (within your other packages) and your users can import as:

import myproject
import packageA.sub_package_in_A.module

etc

Which means you should think hard about @MattAnderson's comment, but if you want it to appear as a separately-distributable package, it needs to be in the top directory.

Note this doesn't stop you (or your users) from doing an:

import packageA.sub_package_in_A as sub_package_in_A

but it does stop you from allowing:

import sub_package_in_A

directly.

Community
  • 1
  • 1
Ethan Coon
  • 751
  • 5
  • 16
  • Good discussion in that link. Thanks. – Steve Sawyer Jun 19 '13 at 02:18
  • This approach seems to require that the entry-point of the application (os.getcwd()) is the top-level directory: `myproject-01`. This is necessary to resolve the fully-qualified imports. This was a worthwhile compromise for me, so thanks. – Eric McLachlan Oct 21 '19 at 09:47
3

I think that this is the best reference for creating a distributable python package:

link removed as it leads to a hacked site.

also, don't feel that you need to nest everything under a single directory. You can do things like 

platform/
    core/
        coremodule
    api/
        apimodule

and then do things like from platform.core import coremodule, etc.

Community
  • 1
  • 1
Cameron Sparr
  • 3,925
  • 2
  • 22
  • 31
3

...it seems that there is an assumption that all referenced packages are below the top-level project folder, not collateral to it.

That's mainly because the current working directory is the first entry in sys.path by default, which makes it very convenient to import modules and packages below that directory.

If you remove it, you can't even import stuff from the current working directory...

$ touch foo.py
$ python
>>> import sys
>>> del sys.path[0]
>>> import foo
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named foo

The thought also occurred to me that perhaps the correct approach is to develop common/framework modules in a separate project, and once tested, deploy those to each developer's environment by installing to the site-packages folder.

It's not really a major issue for development. If you're using version control, and all developers check out the source tree in the same structure, you can easily employ relative path hacks to ensure the code works correctly without having to mess around with environment variables or symbolic links.

However, that also raises questions re distribution.

This is where things can get a bit more complicated, but only if you're planning to release libraries independently of the projects which use them, and/or having multiple project installers share the same libraries. It that's the case, take a look at distutils.

If not, you can simply employ the same relative path hacks used in development to ensure you project works "out of the box".

Community
  • 1
  • 1
Aya
  • 39,884
  • 6
  • 55
  • 55