Importing Python scripts iteratively from relative directories

Question

I have a script called test.py, with the following code (I have simplified things significantly):

from foo import Bar

bar = Bar()
result = bar.do_something()

But I don't just have a single script called foo. I have many scripts called foo, organised in the following directory structure:

└── project
    ├── code
    │   ├── test.py
    └── scripts
        ├── script_1
            └── foo.py
        ├── script_2
            └── foo.py
        ├── script_3
            └── foo.py
        ├── script_4
            └── foo.py
        ├── script_5
            └── foo.py

Each foo.py has a slightly different implementation of something. And what I want to do with test.py, is to test all the scripts out, by importing each one and running some tests on it. Below is some code for this (*'s indicate pseudo-code)

*Get all script directories*
*For each directory in script directories:*
    *import foo.py from this directory*

    bar = Bar()
    result = bar.do_something()

    *Save the result for this directory*

How can I do this? In particular, how can I iteratively import scripts, e.g., *import foo.py from this directory*?

You may need to create an __init__.py file under the project directory to specify which script you are using. Here are some detailed explanations: https://stackoverflow.com/questions/448271/what-is-init-py-for — Han Wang, Nov 25 '19 at 19:36
You must create dunder __init__.py file under the every project directory to access relativly to modules. — Mostafa Babaii, Nov 25 '19 at 19:44

Nathan · Answer 1 · 2019-11-25T20:09:11.530

1

I suggest that you change your scripts to create a package for them, as illustrated here. Then you can simply access each script individually as:

import scripts.script_1.foo

or

from scripts.script_1 import foo

Iterative Importing:

To iterate over the folders and import them, you may use "importlib" library of python. You will need to use "import_module" function from this library. That being said, you still need to include __init__.py in every directory. An example of importing a module using this function is illustrated here.

edited Nov 25 '19 at 20:09

answered Nov 25 '19 at 19:38

Nathan

471
2
12

Thanks. I have tried this, but in your example, I would have to manually write down the names of the script directories (script_1, script_2, script_3 ...). Instead, I would like to be able to create a list of directories, then loop through them all and import "foo" on each loop. How can I do this? I tried create the list of directories as strings, but this did not work. – Karnivaurus Nov 25 '19 at 19:53
I see, then you need to use the importlib library of python to iterate over your folders. However, this library is intended for testing only. You can find information about it here: https://docs.python.org/3/library/importlib.html#importlib.import_module – Nathan Nov 25 '19 at 20:02
1

@Karnivaurus I added more information on how to achieve an iterative importing to my original answer. I hope that helps. – Nathan Nov 25 '19 at 20:10

score 1 · Answer 2 · answered Nov 25 '19 at 20:38

I had to do a couple of different things, but the resulting code snippet looks like this:

import os, sys, importlib

# directory of your script folder
scripts_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))+"/../scripts/")


for root, dirs, files in os.walk(scripts_dir):
    # traverse the folder and find ones that has foo.py
    if 'foo.py' in files:
        sys.path.append(root)
        out = importlib.import_module('foo', package=root)

        # call the function in foo.py.  
        #In this case, I assumed there is a function called test 
        met = getattr(out, 'test')
        print(met())

        # clean up the path and imported modules
        sys.path.remove(root)
        del sys.modules['foo']

Importing Python scripts iteratively from relative directories

2 Answers2