Proper ways to set the path of my app in Python

Question

I have a question in how to properly create a path in Python (Python 3.x).

I developed a small scraping app in Python with the following directory structure.

root
├── Dockerfile
├── README.md
├── tox.ini 
├── src
│   └── myapp
│       ├── __init__.py
│       ├── do_something.py
│       └── do_something_else.py
└── tests
    ├── __init__.py
    ├──  test_do_something.py
    └──  test_do_something_else.py

When I want to run my code, I can go to the src directory and do with python do_something.py But, because do_something.py has an import statement from do_something_else.py, it fails like:

Traceback (most recent call last):
  File "src/myapp/do_something.py", line 1, in <module>
    from src.myapp.do_something_else import do_it
ModuleNotFoundError: No module named 'src'

So, I eventually decided to use the following command to specify the python path: PYTHONPATH=../../ python do_something.py to make sure that the path is seen.

But, what are the better ways to feed the path so that my app can run? I want to know this because when I run pytest via tox, the directory that I would run the command tox would be at the root so that tox.ini is seen by tox package. If I do that, then I most likely run into a similar problem due to the Python path not properly set.

Questions I want to ask specifically are:

1. where should I run my main code when creating my own project like this? root as like python src/myapp/do_something.py? Or, go to the src/myapp directory and run like python do_something.py?
2. once, the directory where I should execute my program is determined, what is the correct way to import modules from other py file? Is it ok to use from src.myapp.do_something_else import do_it (this means I must add path from src directory)? Or, different way to import?

3. What are ways I can have my Python recognize the path? I am aware there are several ways to make the pass accessible as below:

   a. write export PYTHONPATH=<path_of_my_choice>:$PYTHONPATH to make the path accessible temporarily, or write that line in my .bashrc to make it permanent (but it's hard to reproduce when I want to automate creating Python environment via ansible or other automation tools)

   b. write import sys; sys.path.append(<root>) to have the root as an accessible path

   c. use pytest-pythonpath package (but this is not really a generic answer)

Thank you so much for your inputs!

my environment

OS: MacOS and Amazon Linux 2
Python Version: 3.7
Dependency in Python: pytest, tox

Answer 1

I would suggest to use setup.py to make this a python package. Then you can install it in development mode python setup.py develop. This way it will be available in your python environment w/o needing to specify the PYTHONPATH.

For testing, you can simply install the package python setup.py install.

Hope that helps.

Answer 2

Two simple steps should make it happen. Python experts can comment if this is a good way to do it (especially going by the concluding caution raised towards the end of this post).

I would have done it like below.

First I would have put a "__init__.py" in root so that hierarchy looks like below. This way python will treat the folder as a package.

root
├── Dockerfile
├── README.md
├── tox.ini 
├── __init__.py
├── src
│   └── myapp
│       ├── __init__.py
│       ├── do_something.py
│       └── do_something_else.py
└── tests
    ├── __init__.py
    ├──  test_do_something.py
    └──  test_do_something_else.py

Then in "do_something.py", I would have added these lines at the top. In the second line please put the full path to the "root" directory.

import sys
sys.path += ['/home/SomeUserName/SomeFolderPath/root']
from  src.myapp.do_something_else import do_it

Please note that the second line will essentially modify the sys.path by adding the root folder path (I guess until the interpreter quits). If this is not what you can afford then I am sorry.