Again, relative import in Python + issue with sudo

Question

I have read plenty of documentation (including this and linked references) on this, but it is simply too difficult for my simple mind: I cannot understand the logics of python import and I usually waste plenty of time in random attempts till I reach a working permutation of settings and commands. May be this is due to the fact that I usually use PyCharm, where everything magically work. Now I am using Visual Studio Code on a remote machine and I need to ask here since I have wasted double the time I usually spend on this without reaching a permutation that works.

Using python 3 on linux (remote machine). The python interpreter is configured with a virtual environment and it does not correspond to the system level one.

I have this project. Its folder structure is mirrored in linux filesystem, i.e., prj, src, commonn, etc. are all folders.

prj
 |- src
 |  some py files
 |  |- common/
 |     - common1.py
 |     - common2.py
 |  |- pipelines/
 |     - main_pipeline1.py  (<- file prefixed with main_ have a __main__ entry point)
 |     - main_pipeline2.py
 |  | - other py module 
 |  | - other py module, ... and others - some of these modules use common
 |- data/ ...
 |- doc/ ...

In pipeline1.py, I have: import common.common1. I corrected this

In what follows $[folder] corresponds to the bash prompt, so $ stands for normal user and folder is the current folder.

When I run pipeline1.py as normal user (on the remote machine), first I get an error:

$[prj/src] python pipeline/pipeline1.py
ModuleNotFoundError: No module named 'common'

In order to have it working I need to add the current folder to PYTHONPATH (that is empty). So

$[prj/src] PYTHONPATH=.
$[prj/src] python pipeline/pipeline1.py

works.

However, the previous script writes in a disk that requires root access, so the previous command needs to be run with sudo. I cannot find a way to run it using sudo:

I tried (after reading, among others, this):

$[prj/src] sudo python pipeline/pipeline1.py
$[prj/src] sudo /path/to/env/bin/python pipeline/pipeline1.py
$[prj/src] sudo -E /path/to/env/bin/python pipeline/pipeline1.py

they all fail, all but the first because python cannot find the module common. Even if I asked to keep the environment with -E (so PYTHONPATH should be kept) the import fails. All the other imports from the virtual environment (that occur before the import common) do not fail.

In the future I need to give the code to a sys admin that might possibly not have any specific knowledge of python: I cannot ask him to set PYTHONPATH check this, check that.

In this case, how should I organize my code to have the import common (or any other module I write) succeed? Do I really need to add PYTHONPATH=. every time?

Is there any kind soul willing to help me? Beer after the pandemic is over.

I made a correction:

import common.common1.py --> common.common1

Answer 1

Add this to the start of pipeline1.py:

import sys
import os
sys.path.append(os.path.realpath(os.path.dirname(__file__), "/.."))
import common.common1

Answer 2

I'm assuming Linux and also that the Python software to be distributed has the setup.py.

Short answer: no, you don't have to modify the PYTHONPATH or sys.path

---

• Create a virtual env (say /opt/myprog) as usual.

• Activate it and install your package (say mypkg) and all its dependencies.

• Put all executable scripts to the bin subdirectory of the virtual env and make sure they start with #!/opt/myprog/bin/python3 shebang line. With correct setup.py this will hapen automatically during installation, see scripts. The scripts will be able to normally import the installed package import mypkg or its parts from mypkg import ...

• Finally symlink the scripts to a directory in users' PATH e.g. to /usr/local/bin. This must be done manually and only once unless you add or rename a script.

Projects installed this way can be normally upgraded (with pip inside an activated environemnt) and the scripts can be normally invoked from the command line.

Answer 3

Based on the structure and your comments, my guess is that you are not trying to pip-install this project as a proper Python package. Rather, it is just a directory with some scripts and modules you want to use. If so, you have at least a couple of options.

First, don't muck around with PYTHONPATH or modifying sys.path. That is almost always a worse approach.

The basic rule for Python importing: the root directory for the purpose of finding packages is the directory of the script/file used to invoke Python. (Ignoring built-ins and packages that have been formally installed, of course.)

Maybe the easiest solution is to move common under pipelines (and optionally make it a package by creating __init__.py inside of it). If you follow the logic of the basic rule, you'll understand why this works (and it won't be affected by full-path vs relative-path issues when invoking python).

src/
    pipelines/
        common/
            __init__.py       # Optional for Python 3.3+
            common1.py
            common2.py
        main_pipeline1.py
        main_pipeline2.py

Another approach is to create simple runner script at the top level. The runner imports the pipelines, selects the right one (based command-line argument or some other configuration), and executes its top-level code (eg, its main()). If the pipelines are not well organized for that type of importing and execution, this approach is quite a bit harder.

src/
    common/
        __init__.py           # Optional
        common1.py
        common2.py
    pipelines/
        __init__.py           # Optional
        main_pipeline1.py
        main_pipeline2.py
    runner.py

Separate issue: import modules, not files.

import common.common1.py   # No
import common.common1      # Yes