Python Imports failing. Relative imports, package recognition, __init__.py , __package__, __all__

Question

I've been reading a lot of threads, the PEP articles related to this problem, which are around 4 of them, but none of them gives a clear idea on some points and I still can't do relative imports.

In fact, the contents of my main package is not listed at all.

REEDITION. I modified all the post, it was too complex and questions were a lot.

In C:/test/ I have this package:

Package/ (FOLDER 2)
    __init__.py
    moduleA.py
    moduleB.py

• moduleA imports moduleB, and viceversa.
• __init__.py is empty

My process:

1. I add C:/test/ to sys.path.
2. import Package (WORKS)
3. dir(Package) Doesn't list any module inside Package.
4. Package is: <module ‘Package’ from C:/test/Package/_init_.py>
5. __file__ is the init file under Package
6. __name__ is Package
7. __package__ is an empty string
8. __path__ is C:/test/Package

---

TEST 1 - Version 1: in moduleA I have from Package import moduleB

I get this:

>>> import Package.moduleA
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:/test\Package\moduleA.py", line
    from Package import moduleB
  File "C:/test\Package\moduleB.py", line
    from Package import moduleA
ImportError: cannot import name moduleA

It doesn't work because moduleA is not part of Package. So Package is not recognized as a package?

---

TEST 1 - Version 2: in moduleA I have from . import moduleB

Doesn't work, same error

---

TEST 1 - Version 3: in moduleA I have import Package.moduleB

It works.

After that, running:

>>> dir(Package.moduleB)
['Package', '__builtins__', '__doc__', '__file__', '__name__', '__package__']
>>> Package.moduleB.Package
<module 'Package' from 'C:/prueba\Package\__init__.py'>
>>> dir(Package)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'moduleA', 'moduleB']

So now, Package.moduleB has Package as a variable And surprisingly, Package how looks like a proper package, and contains both modules.

In fact, doing any of imports in Version 1 and Version 2 work now, because now moduleA and moduleB are part of Package.

---

Questions:

1) Why is Package not recognized as a package? Is it? Shouldn't it contain all submodules?

2) Why running import Package.moduleA generates Package inside moduleA?

3) Why running import Package.moduleA adds moduleA to Package and it wasn't there before?

4) Does an empty __init__.py file, and a NON empty __init__.py file affect to this at all?

5) Does defining an __all__ variable containing ['moduleA', 'moduleB'] would do anything here?

6) How can I make the init file to load both submodules? Should I do import Package.moduleA and Package.moduleB inside?... can't I do it relatively like import .moduleA as moduleA or something like that? (What If the name of Package changes?)

7) Does that empty string on the package variable affect? We should change it's contents if we want it to recognize itself... __package__ should be the same as __name__, or that's what the PEPs say. But doing this didn't work:

if __name__ == "__main__" and __package__ is None:
    __package__ = "Package"

Answer 1

This is a circular dependency issue. It's very similar to this question, but differs in that you are trying import a module from a package, rather than a class from a module. The best solution in this case is to rethink your code such that a circular dependency is not required. Move any common functions or classes out to a third module in the package.

Worrying about __package__ is a red herring. The python import system will set this appropriately when your package becomes a proper package.

The problem is that moduleA and moduleB are only placed in package once they have been successfully imported. However, since both moduleA and moduleB are in the process of being imported they cannot see each other in package. The absolute import partially solves the problem as you bypass the relative import machinery. However, should your modules require parts of each other during initialisation then the program will fail.

The following code would work if the var = ... line was removed.

package.moduleA

import package.moduleB
def func():
    return 1

package.moduleB

import package.moduleA
var = package.moduleA.func() # error, can't find moduleA

Example of a broken package

package.moduleA

from . import moduleB
def depends_on_y():
    return moduleB.y()
def x():
    return "x"

package.moduleB

from . import moduleA
def depends_on_x():
    return moduleA.x()
def y():
    return "y"

Example of extracting common parts into separate module in package

package.common

def x():
    return "x"
def y():
    return "y"

package.moduleA

from .common import y
def depends_on_y():
    return y()

package.moduleB

from .common import x
def depends_on_x():
    return x()

Answer 2

This is a Python bug that exists in Python versions prior to 3.5. See issue 992389 where it was discussed (for many years) and issue 17636 where the common cases of the issue were fixed.

After the fix in Python 3.5, an explicit relative import like from . import moduleA from within a module within the package Package will check in sys.modules for Package.moduleA before giving up if moduleA isn't present yet in Package. Since the module object is added to sys.modules before it starts loading, but is not added to Package.__dict__ until after the loading is complete, this usually fixes the problem.

There can still be a problem with circular imports using from package import *, but in issue 23447 (which I contributed a patch for), it was decided that fixing that even more obscure corner case was not worth the extra code complexity.

Circular imports are usually a sign of bad design. You should probably either refactor the bits of the code that depend on each other out into a single utility module that both of the other modules import, or you should combine both of the two separate modules into a single one.