36

I was working the following example from Doug Hellmann tutorial on multiprocessing:

import multiprocessing

def worker():
    """worker function"""
    print 'Worker'
    return

if __name__ == '__main__':
    jobs = []
    for i in range(5):
        p = multiprocessing.Process(target=worker)
        jobs.append(p)
        p.start()

When I tried to run it outside the if statement:

import multiprocessing

def worker():
    """worker function"""
    print 'Worker'
    return

jobs = []
for i in range(5):
    p = multiprocessing.Process(target=worker)
    jobs.append(p)
    p.start()

It started spawning processes non-stop, and the only way to stop it was reboot!

Why would that happen? Why it did not generate 5 processes and exit? Why do I need the if statement?

Ηλίας
  • 2,560
  • 4
  • 30
  • 44

3 Answers3

47

On Windows there is no fork() routine, so multiprocessing imports the current module to get access to the worker function. Without the if statement the child process starts its own children and so on.

Adam Parkin
  • 17,891
  • 17
  • 66
  • 87
Denis Otkidach
  • 32,032
  • 8
  • 79
  • 100
  • 4
    It is interesting to know now, after it cost me 2 reboots ;) – Ηλίας Apr 23 '10 at 10:06
  • How would you stop this once it has started? Killing the process in task manager does not seem to affect it. – Ηλίας Apr 23 '10 at 10:14
  • 1
    There are limits and killall in POSIX systems, but I don't know solution for Windows. – Denis Otkidach Apr 23 '10 at 10:26
  • 5
    The solution is to reboot :-) If you are quick you can use a warm reboot within the first second. After that only a cold reboot works, since the OS completely freezes. – nikow Apr 23 '10 at 10:32
  • The number of seconds depends on how many you spawn in the loop "for i in range(5):" . The number 5 seem to give about a second to warm reboot. Without it, you can easily kill it. – Ηλίας Apr 23 '10 at 11:45
  • @nikow: Assuming you caught it within the first second, I wonder if "taskkill /F /IM python.exe" would kill it. I don't feel like rebooting, so not going to test this. – Brian Sep 03 '10 at 19:28
10

Note that the documentation mentions that you need the if statement on windows (here).

However, the documentation doesn't say that this kills your machine almost instantly, requiring a reboot. So this can be quite confusing, especially if the use of multiprocessing happens in some function deep inside the code. No matter how deeply hidden it is, you still need the if check in the main program file. This pretty much rules out using multiprocessing in any kind of library.

multiprocessing in general seems a bit rough. It might have the interface of the thread interface, but there is just no simple way around the GIL.

For more complex parallelization problems I would also look at the subprocess module or some other libraries (like mpi4py or Parallel Python).

nikow
  • 21,190
  • 7
  • 49
  • 70
  • Sorry, I didn't find any really simple ones (there is a PyMOTW article for example). Basically you create Python processes running your worker script. You can send/receive data by using the stdin/stdout of these processes (e.g., sending objects in pickled form). – nikow Apr 23 '10 at 11:07
  • Note that multiprocessing has its uses, and is still the most simple option if you can get it to work for your problem. But if it doesn't work out for you then using subprocess isn't that much extra work (maybe a hundred lines of code) and it gives you more options. – nikow Apr 23 '10 at 11:10
  • 1
    This recipe gives an effective usage option for the subprocess package: http://code.activestate.com/recipes/577045/ – Noctis Skytower Apr 23 '10 at 11:25
  • @Noctis: That links leads to a multiprocessing example. – nikow Apr 23 '10 at 12:09
  • It's documented, as heavily as I felt comfortable without adding flashing banners. Windows doesn't have fork(). If you have a suggestion for improving it, or patching it, feel free to contact me, file a bug, etc. I'm always happy to take user suggestions. Also note; internally, multiprocessing uses a Popen class similar to subprocess (Lib/multiprocessing/forking.py). It *is* sugar on top of raw subprocesses. – jnoller Apr 23 '10 at 18:51
  • @jnoller: Maybe one could tell explicitly that not having the `if` will often bomb the machine? Before I first used `multiprocessing` I read the docs, forgot about the `if`, and was then confused because instead of getting some exception my computer just froze. Did you look at Parallel Python? They also use some magic for convenience, but less aggressively and therefore don't have this problem. I feel that multiprocessing tries too hard to hide the complexities, because this just doesn't work and often fires back. – nikow Apr 23 '10 at 23:05
  • @jnoeller: I also get a little mad when `multiprocessing` is often sold as the solution to the GIL problem. That way `multiprocessing` was hyped as something that it is clearly not (but of course that isn't your fault). Anyway, I actually do use `multiprocessing` in one case where it does work well (and saves me the `subprocess` boilerplate code), so thank you for this module :-) – nikow Apr 23 '10 at 23:10
4

I don't know about multiprocessing, but I suspect that it spawns child processes that have a different __name__ global. By removing the test, you are making every child start the spawning process again.

Marcelo Cantos
  • 181,030
  • 38
  • 327
  • 365