2

I'd like to be able to use multiprocessing in a module I'm developing, but in Windows I need to ensure that the user wraps their entry point in an if __name__ == "__main__" or else the user will experience a multiprocessing bomb.

Is it possible to detect that multiprocessing bombing may be underway inside a module and terminate? Or even better, avoid a multiprocessing bomb in the first place while still being able to start a multiprocessing.Process inside my module?

Rich
  • 12,068
  • 9
  • 62
  • 94
  • 1
    The answer may vary depending on the implementation but for cPython you can use instrumentation hooks in order to achieve this. Possibly it involves raising and catching an exception in order to inspect the stack frame. – Paulo Scardine May 08 '18 at 21:59
  • By the time control reaches your code in the worker (if it reaches your code at all), the worker may have already done irreversible damage. `multiprocessing` is really confusing and dangerous overall, and this is far from the only problem you'll encounter if you try to use it. – user2357112 May 08 '18 at 22:05
  • 1
    Flesh it out more, and you may not have "a problem" at all here. If it's _your_ module starting `mp` gimmicks, it's _your_ module that will be imported in worker processes, not the main program the user happens to be running. So long as you don't start `mp` gimmicks in your module's global code (e.g., you start them inside a module function instead), worker processes importing your module will be harmless. – Tim Peters May 08 '18 at 22:31
  • 1
    @TimPeters: That doesn't sound right. Doesn't the `spawn` start-method unconditionally try to import whatever the master's `__main__` was, even when used from a different module? That definitely happens when I test it (although I'm currently stuck testing on Linux, and Linux `spawn`-start-method isn't quite the same as Windows). – user2357112 May 08 '18 at 23:00
  • @user2357112, indeed, it appears too complicated to explain - or perhaps to understand ;-) Works fine for me on Windows from an interactive shell (with `-i mymain.py`) then running a function from the `mp`-using module, but not if that interactive input is put _in_ `mymain.py`. I'm in too many other rat-holes right now to care enough to pursue it. It's all yours ;-) – Tim Peters May 08 '18 at 23:59
  • @user2357112 and @TimPeters, thanks for checking on this. I can verify that on Windows a `main.py` that lacks an `if __name__ == '__main__'` will infinitely attempt to load processes. – Rich May 09 '18 at 18:02

0 Answers0