Class is in sys.modules but cannot be used. Why?

Question

I have a file MySQL.py which contains a class MySQL defined like so:

class MySQL:
    ... all stuff that is not important here

In other file (test.py), which is in the same directory I do a conditional loading of this MySQL class. By this conditional loading I mean, that I load it in case it has not been loaded yet. To check it, I use sys.modules like so:

print("MySQL" not in sys.modules)
if "MySQL" not in sys.modules:
    from MySQL import MySQL
    print("Loaded it")
    print("MySQL" not in sys.modules)
return MySQL()

As you can see, I have some print's for debugging purposes. And this is what I get in the console, when I run this file:

$ python3 test.py
True
Loaded it
False
Traceback ...
...
UnboundLocalError: local variable 'MySQL' referenced before assignment

It is really interesting, because in the console we see, that at first the module is not loaded (print("MySQL" not in sys.modules) => True), then we see that it gets loaded, but finally for some crazy reason Python does not see this class. PS. I should add, that if I import at the very start of my file (before all other code, then everything is ok).

EDIT

I think, I got it, The whole reason of all troubles is that the way I do import puts my class to sys.modules, but at the same time it puts it to the local namespace of the function and not the global namespace of the module. That's it.

Why don't you just import at the very start to make everything okay, like you're supposed to anyway? — TigerhawkT3, Oct 19 '15 at 11:03
Is that code in `test.py` inside a function? If so, you have not imported `MySQL` the second time, so its not present in the local or global namespace, causing the issue. — Anand S Kumar, Oct 19 '15 at 11:03
*Why* are you doing *"conditional loading"*? If it is already loaded, then `import` just becomes a simple lookup in `sys.modules`. — jonrsharpe, Oct 19 '15 at 11:04
@Anand S Kumar. Yes, that code is inside a function and returning statement is also inside that very function — Jacobian, Oct 19 '15 at 11:04
Well, then like Tigerhawk says, what is the problem with loading it at the start of the script? — Anand S Kumar, Oct 19 '15 at 11:05
@Jacobian ...yes. See e.g. https://docs.python.org/2/tutorial/modules.html#more-on-modules — jonrsharpe, Oct 19 '15 at 11:06
@Anand S Kumar. The catch is, that file does a kind of instantiation. It makes instances of different classes (some of them may be quite "heavy"). So, it is not reasonable to load all that stuff at once — Jacobian, Oct 19 '15 at 11:08
@jonrsharpe. I was not sure of that, since I've seen some discussions and threads here at stackoverflow devoted to the problem of importing. The question was - how to import a file, in case it has not been imported yet. And some benchmarks showed, that if to do `import` without any conditioning, then there is some overhead. But I'm not sure of that — Jacobian, Oct 19 '15 at 11:11
@Jacobian Yes, as jonrhsarpe said, it would only be really imported once and then stored in `sys.modules` , for all subsequent imports it would be loaded from `sys.modules` . Which thread says that you need to import conditionally by checking inside `sys.modules` ? — Anand S Kumar, Oct 19 '15 at 11:13
You may check this thread - http://stackoverflow.com/questions/5027352/how-to-test-if-one-python-module-has-been-imported — Jacobian, Oct 19 '15 at 11:14
["Imports are always put at the top of the file, just after any module comments and docstrings, and before module globals and constants."](https://www.python.org/dev/peps/pep-0008/#imports) — TigerhawkT3, Oct 19 '15 at 11:14
@TigerhawkT3. Even though it is documentation, I'm not sure that we should always obey this rule. — Jacobian, Oct 19 '15 at 11:16
@Anand S Kumar. As you can see from that thread, one guy suggests just doing import and forget about it, whereas another guy, says that it is not totally correct and that there is still some overhead. — Jacobian, Oct 19 '15 at 11:18
Well, given that in this case you can obey it and get working code that's short and easy to read, or messy code that doesn't work... — TigerhawkT3, Oct 19 '15 at 11:18
@TigerhawkT3. The code does not become messy, when we put importing statements just before creating instances from imported classes. — Jacobian, Oct 19 '15 at 11:20
@TigerhawkT3. What would you choose - import zillions of classes, when only one class is used during the runtime, or import a class, when actually there is a need of that class? — Jacobian, Oct 19 '15 at 11:21
You have a big ol' block of code checking `sys.modules` when you could have a single line with an import. That's a significant increase in complexity. — TigerhawkT3, Oct 19 '15 at 11:23
Did you try the "simple import at the top" method, determine that it was too slow, profile the code, and find that the import statements were the cause? — TigerhawkT3, Oct 19 '15 at 11:23
Not only I tried that, I even said about that in my question in PS block. Please, have a look at the very last phrase of my question. — Jacobian, Oct 19 '15 at 11:25
Seems like people started to vote down, not even providing any arguments for that. — Jacobian, Oct 19 '15 at 11:26
It just says you tried it and it worked. It doesn't say anything about the performance. Did you use it for a bit, decide it was slow, profile the code, and find that the imports were the bottleneck? — TigerhawkT3, Oct 19 '15 at 11:29
All in all, my question is not about performace and bottlenecks at all. It is about a concrete test-case, that shows that a class is in `sys.modules`, but for some reason at the next line it is impossible to use that class, — Jacobian, Oct 19 '15 at 11:43
Unfortunately, your question is an [XY Problem](http://meta.stackexchange.com/questions/66377/what-is-the-xy-problem). You already had clean, working code - don't mess with it unless/until it's necessary. — TigerhawkT3, Oct 19 '15 at 11:54
@TigerhawkT3. Let me ask you a question, if some class `MyClass` is in `sys.module`, what is the reason that Python still does not see it and is unable to create instances? — Jacobian, Oct 19 '15 at 11:58
I don't know, and I don't need to, because I put all my import statements at the top. When my Ikea instructions tell me to thread screw A into nut B, I don't pull out my welding torch. :) — TigerhawkT3, Oct 19 '15 at 12:05
"I put all my import statements at the top". Sounds like I put all my methods in one "God class" and I'm happy with that :) — Jacobian, Oct 19 '15 at 12:10

score 1 · Answer 1 · answered Oct 19 '15 at 11:43

1

Your code doesn't work because if the MySQL module has previously been loaded, you're not importing the class, and so the attempt to call MySQL() on the last line cannot work.

A better approach is to simply do the import unconditionally. Python caches the module once it's been loaded (this is the whole purpose of sys.modules), so if you import it more than once, the heavyweight code you have in it will still only be run once.

That said, it may be a sign of bad design if your module is doing a lot of stuff at the top level. Perhaps you should move some or all of the object creation or whatever inside a function somewhere and call it at an appropriate time.

answered Oct 19 '15 at 11:43

Blckknght

100,903
11
120
169

"Perhaps you should move some or all of the object creation or whatever inside a function somewhere and call it at an appropriate time." - this is actually what I do in my Factory class, that does nothing, but creates instances of different classes. But do you believe it is a bad design, if we put `import` statements before we instantiate particular classes? Will it be a good design, if we import all possible classes at the top, but use only one class during the runtime? – Jacobian Oct 19 '15 at 11:46
@Jacobian - As PEP8 says, yes, it's good design to simply put your imports at the top. – TigerhawkT3 Oct 19 '15 at 11:51
And I cannot understand this idea, that "if the MySQL module has previously been loaded, ..., the attempt to call MySQL() on the last line cannot work.". Why it cannot work, if it has been loaded? – Jacobian Oct 19 '15 at 11:51
@TigerhawkT3. I think there should be another good Design pattern, that says that you should load/import only that code, that you will be actually using. – Jacobian Oct 19 '15 at 11:53
@Jacobian: An import statement does two things: loads the module if necessary (it will only be loaded once, then cached), and put the requested names into your namespace. If you don't get `MySQL` in your namespace, you can't call it. – Blckknght Oct 19 '15 at 11:53
@ Blckknght. " If you don't get MySQL in your namespace, you can't call it." But the whole catch is that MySQL IS in my namespace, at least it is in `sys.modules` (that may be different from what you mean) – Jacobian Oct 19 '15 at 11:55
@Jacobian: `sys.modules` is not your namespace. You should really read the documentation linked by johnrsharpe in a comment on the question, it explains a number of features of Python modules that you don't seem to understand (for instance, that each module has its own namespace). – Blckknght Oct 19 '15 at 11:57
So, I have a file/module with its class and methods. It is a module, because from documentation - "A module can contain executable statements as well as function definitions" and indeed my file contains all that stuff. In that very file in one of the functions I have an `import` statement. Do you mean, that even if I do this `import` and even check it with sys.modules, it still may be possible that imported class is not in the namespace (probably now shrinked to the function where I have a call?) – Jacobian Oct 19 '15 at 12:04
In other words, does the appearance of a class in `sys.modules` guarantee that this class can be used or not? And if not, why? – Jacobian Oct 19 '15 at 12:06
This extended discussion in comments really isn't appropriate for Stack Overflow. If you want interactive tutoring, I'm afraid you will need to somewhere else. – Blckknght Oct 19 '15 at 12:22

Class is in sys.modules but cannot be used. Why?

1 Answers1