Why module cannot be found even if I use __init__.py?

Question

I tried to read how import works and use what is suggested, but I still have questions:

If I cd to project folder and run python ./test/test1.py I get ModuleNotFoundError: No module named 'lib'.

• If I add print(os.getcwd()) it does print the current directory which is project. So, why it cannot find lib folder?
• If I cd to project/test, add import sys sys.path.append('..') to the beginning of test1.py and then run python test1.py it works. But, why I cannot run it from one level above?
• I also tried to add __init__.py to lib and test folders, but I get same error.
• If I try changing to relative path from .lib.lib1 ... and run the test.py from project folder I get ImportError: attempted relative import with no known parent package error

What am I doing wrong?

test1.py

from lib.lib1 import MyClass

Project structure

project
  lib
    lib1.py
    lib2.py
  test
    test1.py # from lib.lib1 import MyClass
   
  main.py 

Answer 1

When you run a script, Python automatically adds (only) "its" directory to the sys.path which is where it looks for other modules and packages. In your case, by running python ./test/test1.py, its directory which is <rest of the path>/test/ is going to be added to the sys.path and that's it.

Now where is the lib directory? It is in the <rest of the path>/project/ directory. But the <rest of the path>/project/ directory is not in the sys.path. Yes it's your current working directory but Python doesn't add your "current directory" to the sys.path. Only the script's directory.

The rule is, Check what is inside your sys.path and see if the module you're importing is accessible through one of the records inside that list. If it's not, you need to explicitly add it.

Why are you facing these kind of issues? Normally you won't(shouldn't?) run the sub modules/packages directly. They supposed to be invoked by the main.py module(except for tests though). In that case the <rest of the path>/project/ directory is in sys.path so that all other submodules which use the absolute import statements starting from the root will work properly.

Side note: In the absence of __init__.py module, your import statements still work because of the Implicit Namespace Packages.

Solution:

Add this at the top of your test1.py:

import sys
sys.path.append("<rest of the path>/project")

And then use this form for import in test1.py:

from lib.lib1 import MyClass

As you see, you have to add it to all sub-modules which you directly run, Another option is to use PYTHONPATH environment variable. So instead of writing sys.path.append...., run your script like:

PYTHONPATH="<rest of the path>/project" python ./test/test1.py