conditionally import module to shadow local implementation

Question

I am writing a Python script where some of the core functionalities can be done by another existing library. Unfortunately, while that library has more features, it is also slower, so I'd like if the user could at runtime select whether they want to use that library or my own fast and simple implementation. Unfortunately I'm stuck at a point where I don't understand some of the workings of Python's module system.

Suppose that my main program was main.py, that the (optional) external module is in module_a.py and that my own fast and simple implementation of module_a together with the actual program code that uses either my own implementation or the one of module_a is in the file module_x.py:

main.py:

import module_x
module_x.test(True)
module_x.test(False)

module_a.py:

class myclass():
    def __init__(self):
        print("i'm myclass in module_a")

module_x.py:

class myclass():
    def __init__(self):
        print("i'm myclass in module_x")

def test(enable_a):
    if enable_a:
        try:
            from module_a import myclass
        except ImportError:
            global myclass
            enable_a = False
    else:
        global myclass
    i = myclass()

When I now execute main.py I get:

$ python3 main.py
i'm myclass in module_a
i'm myclass in module_a

But why is this? If False is passed to test() then the import of the module_a implementation should never happen. Instead it should only see myclass from the local file. Why doesn't it? How do I make test() use the local definition of myclass conditionally?

My solution is supposed to run in Python3 but I see the same effect when I use Python2.7.

Try to comment line module_x.test(True) and run with False at start, you'll see — Sergius, Nov 01 '15 at 08:56
That is not a solution because after having run `module_x.test(True)` only once, `module_x.test(False)` will refuse to work. Why is this and how to fix it? — josch, Nov 01 '15 at 08:59
It's because variable myclass has been created at first run, and never been reassigned or destroyed. Also, it shadows myclass from module_x forever. So, u should rebuild this module with another approach — Sergius, Nov 01 '15 at 09:03
Great, which one? That things work this way seems totally unintuitive to me. — josch, Nov 01 '15 at 09:12
A simple solution is to `import myclass as myclass_slow` and define `class myclass_fast` , then conditionally set `myclass = myclass_fast if use_fast else myclass_slow`. — Terry Jan Reedy, Nov 02 '15 at 02:40
@TerryJanReedy your suggestion worked and is much more readable and needs less code than lemonhead's solution. If you write it down as a proper answer I can accept it and upvote you. — josch, Nov 14 '15 at 07:45

score 1 · Answer 1 · edited May 23 '17 at 12:31

An import statement is permanent within the thread of execution unless it is explicitly undone. Furthermore, once the from ... import statement is executed in this case, it replaces the variable myclass in the global scope (at which point the class it was previously referencing defined in the same file is no longer referenced and can in theory be garbage collected)

So what is happening here is whenever you run test(True) the first time, your myclass in module_x is effectively deleted and replaced with the myclass from module_a. All subsequent calls to test(False) then call global myclass which is effectively a no-op since the global myclass now refers to the one imported from the other class (and besides the global call is unneeded when not changing the global variable from a local scope as explained here).

To work around this, I would strongly suggest encapsulating the desired module-switching behavior in a class that is independent of either module you would like to switch. You can then charge that class with holding a reference to both modules and providing the rest of you client code with the correct one. E.g.

module_a_wrapper.py

import module_x
import module_a

class ModuleAWrapper(object):
    _target_module = module_x # the default

    @classmethod
    def get_module(cls):
        return cls._target_module

def set_module(enable_a):
    if enable_a:
        ModuleAWrapper._target_module = module_a
    else:
        ModuleAWrapper._target_module = module_x

def get_module():
    return ModuleAWrapper.get_module()

main.py:

from module_a_wrapper import set_module, get_module
set_module(True)
get_module().myclass()
set_module(False)
get_module().myclass()

Running:

python main.py

# Outputs:
i'm myclass in module_a
i'm myclass in module_x

You can read more about the guts of the python import system here

score 0 · Accepted Answer · answered Nov 14 '15 at 08:04

The answer by lemonhead properly explains why this effect happens and gives a valid solution.

The general rule seems to be: wherever and however you import a module, it will always replace any variables of the same name from the global scope.

Funnily, when I use the import foo as bar construct, then there must neither be a global variable named foo nor one named bar!

So while lemonhead's solution worked it adds lots of complexity and will lead to my code being much longer because every time I want to get something from either module I have to prefix that call with the getter function.

This solution allows me to solve the problem with a minimal amount of changed code:

module_x.py:

class myclass_fast():
    def __init__(self):
        print("i'm myclass in module_x")

def test(enable_a):
    if enable_a:
        try:
            from module_a import myclass
        except ImportError:
            enable_a = False
            myclass = myclass_fast
    else:
        myclass = myclass_fast
    i = myclass()

So the only thing I changed was to rename the class I had in global scope from myclass to myclass_fast. This way it will not be overwritten anymore by the import of myclass from module_a. Then, on demand, I change the local variable myclass to either be the imported module or myclass_fast.

conditionally import module to shadow local implementation

2 Answers2