1

Consider the following program to load all modules in a directory:

import pkgutil
import os.path
import sys

def load_all(directory):
    for loader, name, ispkg in pkgutil.walk_packages([directory]):
        loader.find_module(name).load_module(name)

if __name__ == '__main__':
    here = os.path.dirname(os.path.realpath(__file__))
    path = os.path.join(here, 'lib')
    sys.path.append(path)
    load_all(path)

Now consider that we have the following files.

lib/magic/imho.py:

"""This is an empty module.
"""

lib/magic/wtf.py:

import magic.imho
print "magic contents:", dir(magic)

lib/magic/__init__.py:

"Empty package init file"

When executing the program above, it output:

magic contents: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

In other words, despite the import magic.imho there is no attribute imho on the package magic which causes any later references to magic.imho to fail.

Running the same code (more or less) directly in python shows a different output, which is the one I expected from running the loading program.

$ PYTHONPATH=lib ipython
Python 2.7.6 (default, Mar 22 2014, 22:59:38) 
Type "copyright", "credits" or "license" for more information.

IPython 1.2.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import magic.wtf
magic contents: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'imho']

Why is there a difference?

Update: The point here is that the symbol magic.imho does not exist inside magic.wft despite the fact that it was explicitly imported. So, what need to be done to the custom load procedure to make it behave correctly and ensure that the symbol is visible after it has been imported in magic.wtf.

Solution using setattr: BartoszKP gave the solution below using an exec to assign the value to the package. Since I do not usually like using exec (for a lot of reasons, injections is the first one) I used setattr and rpartition as follows:

def load_all(directory):
    for loader, name, ispkg in pkgutil.walk_packages([directory]):
        module = loader.find_module(name).load_module(name)
        pkg_name, _, mod_name = name.rpartition('.')
        if pkg_name:
           setattr(sys.modules[pkg], mod_name, module)
Community
  • 1
  • 1
Mats Kindahl
  • 1,863
  • 14
  • 25

1 Answers1

0

As suggested in this answer to mimic more accurately what import is doing you should modify your load_all method as follows:

def load_all(directory):
    for loader, name, ispkg in pkgutil.walk_packages([directory]):
        module = loader.find_module(name).load_module(name)
        exec('%s = module' % name)

This way it works, and wtf.py will have the name properly bounded inside magic. So next time magic is imported it won't be reloaded (because this function puts it in sys.modules already) but it will have all its submodules properly assigned.

This seems to alleviate some quirk in import mechanisms, which I've tried to diagnose below.

The difference effectively comes from the the fact that in the first case modules are loaded in a different order than in the second case. When you add print statements to each module you'll see the following when executing the main script:

loading magic

loading imho

loading wtf

magic contents: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

For the second case:

loading magic

loading wtf

loading imho

magic contents: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'imho']

So, in the first case magic and imho are already loaded when wtf.py executes, and the import statements doesn't reload already loaded modules:

First, if the module already exists in sys.modules (a possibility if the loader is called outside of the import machinery) then it is to use that module for initialization and not a new module

If you change your main script like this for example:

if __name__ == '__main__':
    here = os.path.dirname(os.path.realpath(__file__))
    path = os.path.join(here, 'lib')
    sys.path.append(path)
    import magic.wtf

It works exactly the same as from the interpreter.

Your original script will also work as intended if you modify wtf.py as follows (for the purpose of demonstration):

import sys

try:
    del sys.modules['magic.imho']
except:
    pass

import magic.imho

print "magic contents:", dir(magic)

or like this:

import sys
import magic.imho
magic.imho = sys.modules['magic.imho']

print "magic contents:", dir(magic)

Output:

loading magic

loading imho

loading wtf

loading imho

magic contents: ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'imho']

Note: a module exists in sys.modules if it's loaded and that's different from being imported. In first case it's loaded, but was not imported. So it exists in sys.modules but not in current namespace. So I'm guessing that it was loaded by your loader without imho bound to it (so it's just bare magic module). Afterwards when Python sees import magic.imho it checks that magic and magic.imho are already loaded (can be verified by print sys.modules) and takes this version of magic and binds it to local variable magic.

Related:

Community
  • 1
  • 1
BartoszKP
  • 34,786
  • 15
  • 102
  • 130