Python pass variable to multiprocessing pool

Question

I been trying to figure out this for ages, but wondering if anyone might know how I can pass the s variable to the pool without making it into an argument?

import ctypes
import multiprocessing as mp
import os

def worker1(n):
    k = n*3
    print(k)
    print(s)
    # print(ctypes_int.value)

if __name__ == '__main__':
    # global s
    somelist = [1,2,3]

    # ctypes_int = mp.Value(ctypes.c_wchar_p , "hi")
    s = "TESTING"
    # worker1(1)
    p = mp.Pool(1)
    p.map(worker1,somelist)

This is the error I am getting:

3
6
9
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 44, in mapstar
    return list(map(*args))
  File "C:\Users\light\AppData\Local\Temp\tempCodeRunnerFile.python", line 10, in worker1
    print(s)
NameError: name 's' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\light\AppData\Local\Temp\tempCodeRunnerFile.python", line 21, in <module>
    p.map(worker1,somelist)
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Program Files\Python\Python37\lib\multiprocessing\pool.py", line 657, in get
    raise self._value
NameError: name 's' is not defined

Does this answer your question? [Shared variable in python's multiprocessing](https://stackoverflow.com/questions/17377426/shared-variable-in-pythons-multiprocessing) — Bram Vanroy, Jan 05 '21 at 08:34
um I tried it but it didn't work...maybe I did smth wrong? import ctypes import multiprocessing as mp def worker1(n): k = n*3 print(k) print(s) # print(ctypes_int.value) if __name__ == '__main__': somelist = [1,2,3] # ctypes_int = mp.Value(ctypes.c_wchar_p , "hi") # s = "TESTING" mgr = mp.Manager() s = mgr.Value(ctypes.c_wchar_p , "hi") print(s.value) # worker1(1) p = mp.Pool(1) p.map(worker1,somelist) — user10297899, Jan 05 '21 at 08:51
@user as your comment below, you have 8 variable. Please, present this on the post. — Thuấn Đào Minh, Jan 05 '21 at 08:53

score 2 · Accepted Answer · answered Jan 05 '21 at 08:41

2

You can pass your variable along with each item in somelist:

import multiprocessing as mp

def worker1(p):
    n,s = p
    k = n*3
    print(k)
    print(s)

if __name__ == '__main__':
    somelist = [1,2,3]

    s = "TESTING"
    p = mp.Pool(1)
    p.map(worker1,[(n,s) for n in somelist])

The parameter (n,s) gets passed as p and I unpack it into n,s.

answered Jan 05 '21 at 08:41

quamrana

37,849
12
53
71

um it works, but in my actual script I have maybe 8 other variables, would you know if there are any other methods? Thanks! – user10297899 Jan 05 '21 at 08:48
1

I don't see the problem. Just list them along-side `s`: `[(n,s1,s2,s3,s4,s5,s6,s7,s8) for ...` – quamrana Jan 05 '21 at 08:54
@quamrana, Thank you for supporting. Your answers are very helpful :) – Thuấn Đào Minh Jan 05 '21 at 08:55
There much be a better way. We can't go on making the tuple and unpacking it at the other end. – Hayat Jan 17 '21 at 17:58
Perhaps this [answer](https://stackoverflow.com/a/5442981) might help. – quamrana Jan 17 '21 at 19:04

Python pass variable to multiprocessing pool

1 Answers1