0

I am trying to use some parallel computations in my implementation of machine learning algorithms, using joblib, especially the technique used on this page. The following examples are for me to understand parallelism, I have the same problem as the second example in my ML algorithms.

This example runs on all 4 cores as expected:

from joblib import Parallel, delayed

N_PARAM = 10000
N_TEST_FUN = 10000000

def testfunc(data):
    for _ in range(N_TEST_FUN):
        for i in data:
                i*i

def run(niter=10):
    data = [list(range(N_PARAM)) for _ in range(niter)]
    pool = Parallel(n_jobs=-1, verbose=1, pre_dispatch='all')
    results = pool(delayed(testfunc)(dd) for dd in data)

if __name__ == '__main__':
    run()

While this example only runs on 1:

from joblib import Parallel, delayed

N_PARAM = 10000
N_TEST_FUN = 10000000

def testfunc():
    for _ in range(N_TEST_FUN):
        for i in range(N_PARAM):
            i**2

def run(niter=10):
    pool = Parallel(n_jobs=-1, verbose=1, pre_dispatch="all")
    pool(testfunc() for _ in range(niter))

if __name__ == "__main__":
    run()

which I do not understand at all. Why is that ?

I am Ubuntu 18.10, I use joblib 0.13.2 and python 3.6.8 from an Anaconda distribution.

Robin Vogel
  • 159
  • 9

1 Answers1

0

you can find an answer there:
what-does-the-delayed-function-do-when-used-with-joblib-in-python
The accepted answer is well explained.

From my understanding, your second example the pool(testfunc() for _ in range(niter)) returns the result before the function can be pass to the multiprocess.

RadaKk
  • 198
  • 7