1

I want to reduce the time of my calculations with multiprocessing. I do not understand why with numpy object, the multiprocessing work as bad (or what I am doing wrong).

I compare calculation with in parallel 1 process, 3 processes or 6 processes (with a 6 core CPU -> 12 logical CPU). (All cores are used, I am on windows : I tried this answer but it is useless)

  • With the simple_function (without use of numpy object) the multiprocessing works correctly:

    • average time for 1 process in parallel : 3.4s (100%)
    • average time for 3 process in parallel : 1.3s ( 38%)
    • average time for 6 process in parallel : 0.9s ( 25%)

  • With the numpy_function (without numpy matrix) the multiprocessing is almost useless:

    • average time for 1 process in parallel : 0.18s (100%)
    • average time for 3 process in parallel : 0.12s ( 71%)
    • average time for 6 process in parallel : 0.13s ( 74%)
from multiprocessing import Pool
from time import time
from numpy import ones

def simple_function(x):
    for i in range(int(1e8)):
        x*x


def numpy_function(x):
    m = x*ones((1000, 1000))
    for i in range(int(100)):
        m*2


if __name__ == '__main__':
    parallel_comput = 1

    pool = Pool(12)
    print('simple_function')
    for ii in range(10):
        t0 = time()
        pool.map(simple_function, range(parallel_comput))
        tf = time() - t0
        print(tf)

    print('numpy_function')
    for ii in range(10):
        t0 = time()
        pool.map(numpy_function, range(parallel_comput))
        tf = time() - t0
        print(tf)
  • parallel_comput is set to 1, 3 or 6

How can I improve my code ?

Clément v
  • 123
  • 1
  • 7
  • Numpy itself is already optimized for multi-threads / vectorization. Manual threading wouldn't be much better, if not worse due to additional overhead. – Quang Hoang Jan 27 '21 at 17:02
  • 1
    @Quang Hoang, I do not want to parallelize one function of numpy. I want to do parallel calculations which use numpy matrices and function. – Clément v Jan 28 '21 at 08:10

0 Answers0