8

I'm trying to run some multiprocessing in a Jupyter notebook, using python version 3.7.0. However, even a really simple example seems to hang indefinitely. After reading this answer I tried explicitly calling .close and .join on the pool, but it still hangs. Example code is below, can anyone tell me what's wrong?

import multiprocessing as mp

def fun(x):
    return 2*x

with mp.Pool() as pool:
    args = list(range(10))
    res = pool.map(fun, args)
    pool.close()
    pool.join()
bjarkemoensted
  • 2,557
  • 3
  • 24
  • 37
  • 3
    `multiprocessing` has problems working in an interactive prompt. save your code as a .py file, and run it with the python interpreter. (it can be made to work, but it's much simpler to just run the file rather than execute code interactively) – Aaron Mar 02 '21 at 05:00
  • If you look at the console where you started up Jupyter Notebook you will see a bunch of errors. You need to put function `fun` in an external file, for example, `workers.py` in the same directory as your ipynb file and then add `from workers import fun`. In other words, worker functions need to be imported. – Booboo Mar 05 '21 at 18:54
  • 1
    Does this answer your question? [Jupyter notebook never finishes processing using multiprocessing (Python 3)](https://stackoverflow.com/questions/47313732/jupyter-notebook-never-finishes-processing-using-multiprocessing-python-3) – Booboo Mar 05 '21 at 18:56

2 Answers2

4

Another solution is to use the multiprocess module, which is identical to the multiprocessing library while working with Jupyter.

Raisin
  • 345
  • 1
  • 9
0

For me, it worked the solution proposed by @Booboo.

  • Write your function in an external file
  • import it to your .ipynb file
Eduardo Massieu
  • 31
  • 5