Multiprocessing working in Python but not in iPython

Question

I am running the following code in iPython:

import multiprocessing

def my_function(x):
    """The function you want to compute in parallel."""
    x += 1
    return x


if __name__ == '__main__':
    pool = multiprocessing.Pool()
    results = pool.map(my_function, [1,2,3,4,5,6])
    print(results)

in ipython QT console on Windows. However, the code does not work -- the QT console just freezes up. The issue is specific to iPython (the code above should work for the regular Python 2.7).

Any solution to this?

Works fine on Ubuntu. Must be an issue with windows. Do you paste the code into Ipython? — Padraic Cunningham, May 13 '14 at 21:09
Yes, I just tried it in Spyder IDLE. It works there. Any idea how to make it work in the actual QT console? — user3438258, May 15 '14 at 12:41
I second @PadraicCunningham's comment. Works fine on Linux distributions (I'm using Debian). — lostsoul29, Sep 01 '16 at 21:31

score 21 · Answer 1 · answered May 13 '14 at 21:05

21

From the documentation:

Note

Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter

answered May 13 '14 at 21:05

dano

91,354
19
222
219

Did you copy/paste the code in the QT console, or did someting else? – user3438258 May 14 '14 at 19:50
The FINAL answer. No more effort struggling trying to get multiprocessing.Pool working on Jupyter notebook. – Armando Contestabile Dec 27 '22 at 17:56

score 16 · Answer 2 · answered Jan 02 '20 at 20:42

16

At least in the current version of Jupyter Notebook (the successor of IPython) you can solve this by moving the target function to a separate module, and importing it.

I have no idea why this works, it's rather odd, but it does.

i.e. - in a workers.py file put

def my_function(x):
    """The function you want to compute in parallel."""
    x += 1
    return x

Then in IPython/Jupyter notebook put:

import multiprocessing
import workers

pool = multiprocessing.Pool()
results = pool.map(workers.my_function, [1,2,3,4,5,6])
print(results)

also - the if-main thingy doesn't seem to be needed.

Credit: Gaurav Singhal

answered Jan 02 '20 at 20:42

Maverick Meerkat

5,737
3
47
66

This works but it seems like the performance has no increase because it looks like still running in a single process/thread. – John Zhang May 26 '20 at 12:07
@JohnZhang I get different process id's when I run this (although sometimes I do get the same process). You can just add os.getpid() to the worker.py (make sure to import os). – Maverick Meerkat May 26 '20 at 16:28
In my test, I will get MP if I run it in Python console. However, if I run it in Jupyter, it will be always 1 process. I think it makes sense since one Jupyter notebook is one process in Jupyter architecture (at least on Windows). – John Zhang May 27 '20 at 04:43
2

@JohnZhang have you used pool? I also use windows and I do get multiprocessing. Also it doesn't make sense, as every IDE or even console you use is one process, and they all spawn different processes. – Maverick Meerkat May 27 '20 at 07:54

Multiprocessing working in Python but not in iPython

2 Answers2

Linked