16

To preface, I think I may have figured out how to get this code working (based on Changing module variables after import), but my question is really about why the following behavior occurs so I can understand what to not do in the future.

I have three files. The first is mod1.py:

# mod1.py

import mod2

var1A = None

def func1A():
    global var1
    var1 = 'A'
    mod2.func2()

def func1B():
    global var1
    print var1

if __name__ == '__main__':
    func1A()

Next I have mod2.py:

# mod2.py

import mod1

def func2():
    mod1.func1B()

Finally I have driver.py:

# driver.py

import mod1

if __name__ == '__main__':
    mod1.func1A()

If I execute the command python mod1.py then the output is None. Based on the link I referenced above, it seems that there is some distinction between mod1.py being imported as __main__ and mod1.py being imported from mod2.py. Therefore, I created driver.py. If I execute the command python driver.py then I get the expected output: A. I sort of see the difference, but I don't really see the mechanism or the reason for it. How and why does this happen? It seems counterintuitive that the same module would exist twice. If I execute python mod1.py, would it be possible to access the variables in the __main__ version of mod1.py instead of the variables in the version imported by mod2.py?

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
Brendan
  • 457
  • 1
  • 6
  • 10

2 Answers2

25

The __name__ variable always contains the name of the module, except when the file has been loaded into the interpreter as a script instead. Then that variable is set to the string '__main__' instead.

After all, the script is then run as the main file of the whole program, everything else are modules imported directly or indirectly by that main file. By testing the __name__ variable, you can thus detect if a file has been imported as a module, or was run directly.

Internally, modules are given a namespace dictionary, which is stored as part of the metadata for each module, in sys.modules. The main file, the executed script, is stored in that same structure as '__main__'.

But when you import a file as a module, python first looks in sys.modules to see if that module has already been imported before. So, import mod1 means that we first look in sys.modules for the mod1 module. It'll create a new module structure with a namespace if mod1 isn't there yet.

So, if you both run mod1.py as the main file, and later import it as a python module, it'll get two namespace entries in sys.modules. One as '__main__', then later as 'mod1'. These two namespaces are completely separate. Your global var1 is stored in sys.modules['__main__'], but func1B is looking in sys.modules['mod1'] for var1, where it is None.

But when you use python driver.py, driver.py becomes the '__main__' main file of the program, and mod1 will be imported just once into the sys.modules['mod1'] structure. This time round, func1A stores var1 in the sys.modules['mod1'] structure, and that's what func1B will find.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Would this be OK then? `if __name__ == '__main__': sys.modules['mod1'] = sys.modules['__main__']; func1A()` – Kos Nov 01 '12 at 16:50
  • 2
    @Kos: I would not do that, I think you'll find that a whole lot of assumptions will break. Avoid using modules as scripts instead. – Martijn Pieters Nov 01 '12 at 16:51
  • I have been using `if __name__ == '__main__':` to write a testing routine for each of my modules within the module. Is there a good way to continue this practice, or should I write testing routines as functions then have a driver in a separate file that does nothing but import the module to be testing and call the testing routine? – Brendan Nov 01 '12 at 17:25
  • 1
    @Brendan: convention uses a `tests.py` module or a `tests` package, using `unittests` as the framework. – Martijn Pieters Nov 01 '12 at 17:28
  • Pleae can you briefly explain why global var1 is stored in sys.modules['__main__'], but func1B is looking in sys.modules['mod1'] for var1 – variable Oct 06 '19 at 18:51
  • 1
    @variable: because `mod1.func1B()` was called, so the *copy of the function in `mod1` is being used*. Functions use the globals in their own module object. Note that there is also a `__main__.func1B()` function, which is connected to the `__main__` globals. – Martijn Pieters Oct 06 '19 at 18:56
  • 1
    Ah I think I got it. The issue here is that the value is set in the __main__ namespace. But mod2 function is accessing it from mod1 namespace – variable Oct 06 '19 at 19:12
1

Regarding a practical solution for using a module optionally as main script - supporting consistent cross-imports:

Solution 1:

See e.g. in Python's pdb module, how it is run as a script by importing itself when executing as __main__ (at the end) :

#! /usr/bin/env python
"""A Python debugger."""
# (See pdb.doc for documentation.)
import sys
import linecache

...

# When invoked as main program, invoke the debugger on a script
if __name__ == '__main__':
    import pdb
    pdb.main()

Just I would recommend to reorganize the __main__ startup to the beginning of the script like this:

#! /usr/bin/env python
"""A Python debugger."""
# When invoked as main program, invoke the debugger on a script
import sys
if __name__ == '__main__':        
    ##assert os.path.splitext(os.path.basename(__file__))[0] == 'pdb'
    import pdb
    pdb.main()
    sys.exit(0)

import linecache
...

This way the module body is not executed twice - which is "costly", undesirable and sometimes critical.

Solution 2:

In rarer cases it is desirable to expose the actual script module __main__ even directly as the actual module alias (mod1):

# mod1.py    
import mod2

...

if __name__ == '__main__':
    # use main script directly as cross-importable module 
    _mod = sys.modules['mod1'] = sys.modules[__name__]
    ##_modname = os.path.splitext(os.path.basename(os.path.realpath(__file__)))[0]
    ##_mod = sys.modules[_modname] = sys.modules[__name__]
    func1A()

Known drawbacks:

  • reload(_mod) fails
  • pickle'ed classes would need extra mappings for unpickling (find_global ..)
kxr
  • 4,841
  • 1
  • 49
  • 32