2

I have a python module here /root/python/foo.py. I have a bunch of other modules here in the folder /root/lib/ which looks like this


    lib/
    |
    ├─ module1/
    |  ├─ __init__.py
    |  └─ bar.py
    |
    └─ module2/
       ├─ __init__.py
       └─ bar.py

I would like to import /root/lib/module1 and /root/lib/module2 from foo.py. I would like to not have to add /root/lib/ to the python system path. This stack overflow answer tells you how to use either imp.load_source, importlib.machinery.SourceFileLoader, or the importlib.util class to load a module from a file (depending on the python version). I think these only work if the module is a single file. If I try something like this in Python 3.4

from importlib.machinery import SourceFileLoader
problem_module = SourceFileLoader('test_mod', '/root/lib/module1').load_module()

I get an IsADirectoryError

My question is whether there is a similar way to load a module (given its full path) if it is a directory, without adding the whole lib/ folder to the system path?

Community
  • 1
  • 1
pspencer
  • 664
  • 1
  • 6
  • 9
  • add `'` after module1 – Szabolcs Dombi Jul 17 '16 at 18:51
  • You need to use `sys.path` to reliably import modules. See [this question](http://stackoverflow.com/questions/9066777/howto-import-modules-with-dependencies-in-the-same-absolute-relative-path-with-i). – BrenBarn Jul 17 '16 at 18:52
  • 2
    "I would like to not have to add /root/lib/ to the python system path." - Why not? It is *by far* the easiest way to accomplish this. – Kevin Jul 17 '16 at 18:55
  • I didn't want to add the directory to the system path because I thought I might have hundreds of modules in the library and I thought it would be nicer to target just the folder I wanted. Maybe adding the `lib` folder to the system path isn't as bad as I thought. – pspencer Jul 17 '16 at 19:30

2 Answers2

2

try:

from importlib.machinery import SourceFileLoader
problem_module = SourceFileLoader('test_mod', '/root/lib/module1/__init__.py').load_module()

the __init__.py should take care about the modules in the same package:

add from . import bar to make bar.py part of the package.

Some corrections:

  • module1 is a package not a module.
  • bar.py is a module part of the package module1
Szabolcs Dombi
  • 5,493
  • 3
  • 39
  • 71
  • That sort of worked. I wanted to use module1 as a module name, e.g. I wanted to also import bar.py from module1 once it was loaded. Targeting the __init__.py alone didn't let me do that. – pspencer Jul 17 '16 at 19:31
  • `from . import bar` in `__init__.py` – Szabolcs Dombi Jul 17 '16 at 19:41
  • alternative solution: `from .bar import *` – Szabolcs Dombi Jul 17 '16 at 19:42
  • I didn't want to have to import the over modules through the __init__.py file. I wanted to do something like `from problem_module import bar` but I guess if I had to do this it wouldn't be too bad, I would just have to change the way I work. – pspencer Jul 17 '16 at 19:50
  • so you should replace `__init__.py` with `bar.py` (use formatting `%s.py`) – Szabolcs Dombi Jul 17 '16 at 19:55
  • 1
    This was a huge help to me, thank you! I ran into an issue trying to import a file `foo.bar.py`, and `importlib` interpreted `foo`, `bar`, and `py` as modules. – Tom Oct 05 '18 at 13:58
2

Python does not give us an easy way to load files that cannot be referenced by sys.path, most usual solution will do one of the following things:

  1. Add desired paths to sys.path or
  2. Restructure your modules so that the correct path is already on sys.path

Nearly all other solutions that does not do either will be work arounds (using non-intended methods to get the job done) and some can cause quite a headache.

However python does give us a mechanic that lets us simulate a package that is spread out across folders that are not held on sys.path, you can do this by specifying a __path__ special name in a module:

__path__ = ["/root/lib"]

Put this line in a file called lib.py and place it in the same folder as foo.py to be imported by it (so in root/python/ in your case) then from foo.py you can do this as you would expect:

import lib.module1
#or
from lib import module1

This indicates to python that the .module1 subpackage is located somewhere on the specified __path__ and will be loaded from that directory (or multiple directories) using the intended import mechanisms and keeping your sys.path unaltered.

Tadhg McDonald-Jensen
  • 20,699
  • 5
  • 35
  • 59
  • I think you're right and I might just have to end up adding the `lib` folder to the system.path. But if this is the right way to go then I'm wondering why python lets you import a module if it's just a file using `SourceFileLoader` (in python 3.4). If they had that functionality you'd think it wouldn't matter if it's a file or a directory. – pspencer Jul 17 '16 at 19:35
  • Read the name... `Source` **`File`** `Loader`, you didn't specify a **file**, and loading a single module from source is not as simple as the whole import mechanism so I wouldn't count on it working 100% in any case. – Tadhg McDonald-Jensen Jul 17 '16 at 19:37
  • Yeah, I was just wondering why there wasn't a `SourceDirectoryLoader` method when there is a `SourceFileLoader`. But I guess you're right that it's not as simple as just using the import mechanism. – pspencer Jul 17 '16 at 19:43
  • @pspencer I just found an actual solution using intended mechanisms, I think it is exactly what you need. – Tadhg McDonald-Jensen Jul 17 '16 at 20:17
  • @Tadhq McDonald-Jensen Was the solution you were talking about the edit you made above about the `__path__` special name? That is an really interesting feature of python that I didn't know about but it was exactly what I was looking for. I was looking for a method like SourceFileLoader but for directories. I think what I've decided from this discussion is that there isn't a good pythonic way to do what I want except to just add the `lib` directory to the system path. – pspencer Jul 17 '16 at 22:15