How to handle "flat" relative imports in python 3.6?

Question

Since there are so many questions on relative imports, I will make it as short and sweet as possible. And yes, I've read "Relative imports for the billionth time".

I have a project structure like this:

.
├── Makefile
└── src
    ├── __init__.py
    ├── model
    │   └── train_model.py
    └── preprocessing
        └── process.py

where I want to be able to, as an example, call make preprocessingor make train which then runs either process.pyor train_model.py with

## Make train
train:
    python3 src/model/train_model.py

E.g. modules will always from the top project folder where the Makefile lives.

Now, my problem is that i might have dependencies between different submodules, such as train_model.py and process.py. Specifically, if I try to import processin train_model by using from src.preprocessing import process i get an error ImportError: No module named 'src'. In a similar vein, I've tried from ...preprocessing import process, which gives me another error: SystemError: Parent module '' not loaded, cannot perform relative import.

I use if __name__ == '__main__': at the end of my train_model.py, but I can't seem to figure out, how python uses __name__to find different modules, and if this f**** something up in the process.

score 0 · Accepted Answer · answered Feb 24 '18 at 06:20

0

Use PYTHONPATH. I would do it this way:

Makefile:

export PYTHONPATH=$(abspath src)

train:
    python3 src/model/train_model.py

train_model.py:

from preprocessing import process

Now every import will first look under src. It is not conventional to write from src.preprocessing import process - typically imports are understood to be within some base directory (you wouldn't want to set PYTHONPATH to the directory above src, because it may contain things you don't want to import).

answered Feb 24 '18 at 06:20

John Zwinck

239,568
38
324
436

Hi John, and thanks for the answer. The first bit with exporting the pythonpath in the makefile works perfectly. However, the import `from preprocessing import process` does not work. I simply get `ImportError: No module named 'preprocessing`. Any ideas? – tmo Feb 26 '18 at 13:00
@TMorville: What does `print(sys.path)` tell you if you put it in `train_model.py`? How about `print(os.environ.get('PYTHONPATH'))`? – John Zwinck Feb 26 '18 at 13:16
`print(sys.path)` outputs (amongst others) `/src/model` and not just `/src/`as I am guessed it is supposed to do. When calling `make train` the output from `print(os.environ.get('PYTHONPATH'))` is blank. When calling directly with `python3 src/model/train_model.py` it is `None`. – tmo Feb 26 '18 at 13:34
@TMorville: OK so somehow the environment variable is not being set. Try putting `PYTHONPATH=src python3 src/model/train_model.py` in your Makefile.... – John Zwinck Feb 26 '18 at 13:42
Adding the new path works, putting in `'/src python3 src/model/train_model.py'` in `sys.path`. I am guessing that `PYTHONPATH=src python3 src/model/train_model.py`was a mistake? If I simply add `export PYTHONPATH=src` then `project/src` gets added to PYTHONPATH correctly, and now it seems to work! – tmo Feb 26 '18 at 13:56

How to handle "flat" relative imports in python 3.6?

1 Answers1