2

Say I have the following structure:

app/
   __init__.py
   mod.py

   pkg/
      __init__.py
      submod.py

where the module submod has a relative import to mod, i.e.:

from .. import mod

I understand that if I want to execute submod as a script, I can do the following from app/

python -m pkg.submod

But I would like submod.py to be an executable module that I can call from anywhere in the system with

python /path/to/submod.py

I thought that PEP-366 fixed this, i.e. I thought that adding the following boilerplate code before I do any relative imports in submod:

if __name__ == "__main__" and __package__ is None:
    __package__ = "app.pkg"

I could then go back to the regular python /path/to/submod.py. However, when I do that I get:

SystemError: Parent module 'app' not loaded, cannot perform relative import

Why?

Finally I understand that one solution is to manipulate sys.path in submod so that it can see mod1 and then do the regular import mod1 and avoid relative imports. But as this question shows, this is dangerous because any changes to sys.path in one module propagate to everything else, so in general it is not a good idea to tamper with sys.path.

Is there any way to have either:

  • relative imports with support for regular python /path/to/submod.py calls

or

  • the ability to execute my module with python /path/to/submod.py without having to tamper with sys.path or PYTHONPATH

?

Community
  • 1
  • 1
Amelio Vazquez-Reina
  • 91,494
  • 132
  • 359
  • 564
  • You might want to watch out for including scripts inside a package hierarchy: http://stackoverflow.com/questions/18087122/python-sharing-common-code-among-a-family-of-scripts – theodox Aug 19 '13 at 21:19
  • `from .. import pkg1.mod1` isn't a valid import statement in either 2.6-7 or 3.x; it should give you a `SyntaxError` before you can even get to the problem you're trying to debug. If you change it to `from ..pkg1 import mod1`, you get an import-related `SystemError`, but not the one you're showing. It's hard to debug a problem that can't be reproduced in code that we can't see; can you give us an actual [SSCCE](http://sscce.org) that demonstrates your problem? – abarnert Aug 19 '13 at 21:41
  • Also… which Python version are you using? – abarnert Aug 19 '13 at 21:41
  • Thanks @abarnert - I just updated the OP to answer both of your questions – Amelio Vazquez-Reina Aug 19 '13 at 21:47
  • Are you trying to understand why this doesn't work, looking for the least hacky workaround (there are no non-hacky workarounds) to force it to work, or looking for the best way to do what you're actually trying to do? – abarnert Aug 19 '13 at 21:53
  • 1
    @user815423426: There is no non-hacky way to do exactly _this_… but there's a very easy and completely non-hacky way to do what you probably really need: Split `submod.py` into the actual module, and a script that sits alongside the package that just imports the submodule and calls its "main" function. – abarnert Aug 19 '13 at 21:58
  • @abarnert That's actually a great idea. Maybe you can add that to your answer? That's probably the best way to proceed. – Amelio Vazquez-Reina Aug 19 '13 at 21:59
  • @abarnert By the way, would `main.py` be able to import a `submod.py` that has relative imports even though I will still **not** be adding its parents to `PYTHONPATH`? – Amelio Vazquez-Reina Aug 19 '13 at 22:00
  • @user815423426: OK, I'll edit the answer. – abarnert Aug 19 '13 at 22:05

2 Answers2

3

As PEP 366 explicitly says right after the boilerplate that you copied:

Note that this boilerplate is sufficient only if the top level package is already accessible via sys.path. Additional code that manipulates sys.path would be needed in order for direct execution to work without the top level package already being importable.

So, assuming your code looks like this:

if __name__ == "__main__" and __package__ is None:
    __package__ = "app.pkg"
from .. import mod

You should get an exception like this:

SystemError: Parent module 'app' not loaded, cannot perform relative import

As the error message implies, you need to get app imported somehow before you can do any relative imports. That obviously means you can't relative-import app, so you need to absolute-import it. There are various hacky ways you could do this, or you could do what the PEP suggests and munge sys.path with all of the problems that implies. But otherwise, you can't do this.

This is just one of multiple reasons that trying to run a script from the middle of a package is hard. And that's intentional; as explained in PEP 3122:

Guido views running scripts within a package as an anti-pattern.

There are two ways to accomplish what you want. The first, you've already got working: just run the module as a module instead of a script. (And making that work is what PEP 366 was primarily for.) The other is to split the module and script into two pieces. Which you can do trivially just by writing a wrapper.

For example, let's say submod.py looks like this silly toy example:

if __name__ == "__main__" and __package__ is None:
    __package__ = "app.pkg"
from .. import mod

def main(argv):
    print(argv)

if __name__ == '__main__':
    import sys
    main(sys.argv)

Create a new file, say, sub.py, as a sibling to the package:

import sys
import app.pkg.submod
app.pkg.submod.main(sys.argv)

Now, you can run sub.py from anywhere on the system. As sys.path says:

As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter.

Since that directory is also the directory app is in, you're guaranteed to be able to absolute-import app.pkg.submod, which means that submod will be able to relative-import whatever it wants.

abarnert
  • 354,177
  • 51
  • 601
  • 671
1

The package isn't pkg2; it's app.pkg2. Set __package__ accordingly:

if __name__ = '__main__' and __package__ is None:
    __package__ = 'app.pkg2'

If that's not the problem, then app probably isn't on the path. The best solution I know of is to put it on the path (by moving app or putting its current directory in PYTHONPATH), but if you're looking for a way to let modules in app see each other without being visible to external modules, I don't know of anything. It seems like it could be useful.

user2357112
  • 260,549
  • 28
  • 431
  • 505