p.start() blocking the rest of my loop

Question

I'm trying to run multiple main() functions from a number of modules at the same time. I first tried using threading which worked, but I couldn't't find a way to terminate the processes when the parent process is killed nicely. so I moved onto the multiprocessing module. However for some reason p.start() is blocking and only the first of 3 modules in the loop start.

import os
import signal
import sys
from multiprocessing import Process

jobs = []
for root, dirs, files in os.walk('modules'):
        for dir in dirs:
                print dir
                module = getattr(__import__("modules." + dir), dir)
                func = getattr(module, "main", None)
                if func:
                        p = Process(target=func)
                        jobs.append(p)
                        p.start()
                dirs[:] = []  # don't recurse into directories.

print jobs

Output is:

limitlessLED
[<Process(Process-1, started)>]
Connected to MQTT

don't use `os.walk` that traverses `modules` directory *recursively*. Your code: `__import__("modules." + dir)` doesn't handle it anyway. Use `os.listdir("modules")` instead. — jfs, Dec 29 '13 at 08:32

score 1 · Accepted Answer · edited May 23 '17 at 12:19

1

You're destroying the contents of dirs while you're iterating over it:

   for dir in dirs:
            ...
            dirs[:] = []  # don't recurse into directories.

That's why you're only going around the loop once: you empty dirs at the end of the first iteration of the loop. Or the indentation is screwed up in the code you posted. Hard to guess which from here ;-)

Edit: more

OK, looks like you copied the gist of your code from this answer. You didn't copy the indentation correctly: you indented the dirs[:] = [] line one level too much.

edited May 23 '17 at 12:19

Community

1
1

answered Dec 29 '13 at 03:05

Tim Peters

67,464
13
126
132

ha! So I did. Still getting used to having indenting affecting program flow. Thanks – PythonLearner Dec 29 '13 at 04:19

score 0 · Answer 2 · edited May 23 '17 at 11:49

_{It is posted as an answer rather than a comment, to take advantage of formatting capabilities unavailable for comments}

I'm trying to run multiple main() functions from a number of modules at the same time.

You could use subprocess module, to execute all modules' subdirectories as Python packages in parallel:

import sys
from glob import glob
from subprocess import Popen

processes = [Popen([sys.executable or 'python', '-m', dirname.replace('/', '.')])
             for dirname in glob('modules/*/')]

Popen doesn't block your program so all processes run at the same time. And Popen has .terminate(), .kill() methods that you can call any time you like, to stop the subprocesses.

It expects that packages are executable i.e., there are __main__.py files:

from . import main

main()

p.start() blocking the rest of my loop

2 Answers2