Python imports with __init__.py

Question

No matter how I structure the imports in the code files and in the __init__.py files, I can't seem to get it right for executing the program and running the tests using pytest. How do I need to write the imports when my project structure looks like this:

src/
    __init__.py
    VocableFileWriter.py
    WordListHelper.py
    WordListReader.py
    XLDAttributeValueAdder.py

    exceptions/
        __init__.py
        XMLInvalidException.py
        XMLParseException.py

    gui/
        __init__.py
        GTKSignal.py
        XLDAttributeValueAdderWindow.py

    test/
        __init__.py
        test_XLDAttributeValueAdder.py

    xmlparser/
        __init__.py
        XMLParser.py

Currently I have them like this:

In the __init__.py files I have the imports like this (src/__init__.py):

from src import *
from src.exceptions import *
from src.xmlparser import *

and in a subpackage (src/xmlparser/__init__.py):

from src.xmlparser import *

So I guess those are project-absolute paths to the modules. In the code files themselves I import like this:

import os
import sys
from VocableFileWriter import VocableFileWriter
from XLDAttributeValueAdder import XLDAttributeValueAdder

However, when I execute the code from the directory above src using:

./src/main.py

It tells me that:

Traceback (most recent call last):
File "./src/main.py", line 6, in <module>
    from VocableFileWriter import VocableFileWriter
File "/home/xiaolong/Development/PycharmProjects/xld-attribute-value-adder/src/VocableFileWriter.py", line 2, in <module>
    from XMLInvalidException import XMLInvalidException
ImportError: No module named 'XMLInvalidException'

It used to be a PyCharm project, but I couldn't get it to run with the imports structure PyCharm used when not using PyCharm but running it from the terminal, so I decided I wanted to take the whole import stuff into my own hands. So don't get confused by it being in a PyCharm directory.

I also want to be able to execute the tests using for example:

py.test src/test/test_XLDAttributeValueAdder.py

How do I solve this mess?

Edit#1: I had the program running once, but then the test complained about missing modules, probably because it's in another subdirectory and I tried so many configurations, that I don't know how I had the program running anymore. If possible please add some explanation why a certain structure is correct and works for both, tests and the program itself.

EDIT#2: I've managed to get the tests running now, but now the program doesn't run anymore. I emptied all the __init__.py files and used only project-absolute paths in the code files like this (src/test/test_XLDAttributeValueAdder.py):

from src.VocableFileWriter import VocableFileWriter
from src.WordListHelper import WordListHelper
from src.WordListReader import WordListReader
from src.XLDAttributeValueAdder import XLDAttributeValueAdder
from src.exceptions.XMLInvalidException import XMLInvalidException
from src.exceptions.XMLParseException import XMLParserException
from src.xmlparser.XMLParser import XMLParser
from src.test.path_helper import go_up
from src.test.path_helper import go_in

and in the main.py:

from src.VocableFileWriter import VocableFileWriter
from src.XLDAttributeValueAdder import XLDAttributeValueAdder

the output of ./src/main.py:

./src/main.py 
Traceback (most recent call last):
File "./src/main.py", line 6, in <module>
    from src.VocableFileWriter import VocableFileWriter
ImportError: No module named 'src'

EDIT#3: I've tried the relative import way like this:

• main.py moved one directory up

The structure now looks like this:

main.py

src/
    __init__.py
    VocableFileWriter.py
    WordListHelper.py
    WordListReader.py
    XLDAttributeValueAdder.py

    exceptions/
        __init__.py
        XMLInvalidException.py
        XMLParseException.py

    gui/
        __init__.py
        GTKSignal.py
        XLDAttributeValueAdderWindow.py

    test/
        __init__.py
        test_XLDAttributeValueAdder.py

    xmlparser/
        __init__.py
        XMLParser.py

• __init__.py files empty
• main.py file gets relative

Imports look like this:

import os
import sys
from .src.VocableFileWriter import VocableFileWriter
from .src.XLDAttributeValueAdder import XLDAttributeValueAdder

When I try to run the main.py file:

Traceback (most recent call last):
File "main.py", line 6, in <module>
    from .src.VocableFileWriter import VocableFileWriter
SystemError: Parent module '' not loaded, cannot perform relative import

However, there is a __init__.py file in the same directory as the main.py file, also empty.

Answer 1

You'd better use relative imports.

When doing from src import * in src/__init__.py you'll only import anything you've defined before calling the import. Most likely that's just nothing. If you've defined the __all__ variable, you'll get the submodule from it, but maybe also an AttributeError if a variable hasn't been defined yet.

So instead import the modules you need explicitly where you need them like

from .VocableFileWriter import VocableFileWriter
from .exceptions.XMLInvalidException import XMLInvalidException

or lets say within src/gui/GTKSignal.py

from ..exceptions.XMLInvalidException import XMLInvalidException

Also you could use project-absolute paths like mentioned in your Edit#2.

---

Furthermore you've a problem with your path when calling ./src/main.py. In this case the directory containing src is not in the path. Instead ./src/ is in the path. Therefore python doesn't consider it a package anymore and since there's no other package with that name available, the import fails.

Instead put your main.py module in the same directory like the src package or call it as a module with python -m src.main. Doing so you could also rename it to src/main.py and call the package instad with python -m src.

---

When using the first approach, main should use an absolute import structure. Just think of it as a module you put anywhere on your computer, while your src package is somewhere else, where it can be found by Python, like any other module or package installed by the system. It's not different, when it's located besides the package itself, since the current directory is in sys.path.

main.py

import os
import sys
from src.VocableFileWriter import VocableFileWriter
from src.XLDAttributeValueAdder import XLDAttributeValueAdder