As a simple example, when I run the following code:
from multiprocessing import Pool
class foo(object):
def __init__(self):
self.a = 0
def g(self, x):
return x + self.a
def bar(self):
def f(x):
return self.g(x)**2
# Compute [f(x) for x in range(1000)] in parallel
pool = Pool()
res = pool.map(f, range(1000))
pool.close()
pool.join()
return res
res = foo().bar()
I get the error:
---------------------------------------------------------------------------
PicklingError Traceback (most recent call last)
<ipython-input-7-cbbd76310b60> in <module>()
19 return res
20
---> 21 res = foo().bar()
<ipython-input-7-cbbd76310b60> in bar(self)
13 # Compute [f(x) for x in range(1000)] in parallel
14 pool = Pool()
---> 15 res = pool.map(f, range(1000))
16 pool.close()
17 pool.join()
/Users/jiaseny/anaconda2/lib/python2.7/multiprocessing/pool.pyc in map(self, func, iterable, chunksize)
251 '''
252 assert self._state == RUN
--> 253 return self.map_async(func, iterable, chunksize).get()
254
255 def imap(self, func, iterable, chunksize=1):
/Users/jiaseny/anaconda2/lib/python2.7/multiprocessing/pool.pyc in get(self, timeout)
570 return self._value
571 else:
--> 572 raise self._value
573
574 def _set(self, i, obj):
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
I believe it's because the function f called self.g during computation.
Is there any way to resolve this issue?
One work-around would be to make g a local function inside bar, but in more complicated classes this would be impractical since g would probably also contain some other function self.h and so on...
