39

Am getting this error when using the pool.map(funct, iterable):

AttributeError: __exit__

No Explanation, only stack trace to the pool.py file within the module.

using in this way:

with Pool(processes=2) as pool:
   pool.map(myFunction, mylist)
   pool.map(myfunction2, mylist2)

I suspect there could be a problem with the picklability (python needs to pickle, or transform list data into byte stream) yet I'm not sure if this is true or if it is how to debug.

EDIT: new format of code that produces this error :

def governingFunct(list):
    #some tasks
    def myFunction():
         # function contents
    with closing(Pool(processes=2)) as pool:
         pool.map(myFunction, sublist)
         pool.map(myFunction2, sublist2)

ERROR PRODUCED:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed
sidewaiise
  • 1,445
  • 3
  • 16
  • 27
  • what do you mean by `while .. as`? did you mean `with`? – Elisha Sep 22 '14 at 07:20
  • Do include the *full traceback* for errors, and your actual code. `while` is an obvious syntax error here, for example, but from the exception it is clear you used `with` really. – Martijn Pieters Sep 22 '14 at 07:32

2 Answers2

64

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers. You cannot use them in a with statement. Only in Python 3.3 and up can you use them as such. From the Python 3 multiprocessing.Pool() documentation:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

For earlier Python versions, you could use contextlib.closing(), but take into account this'll call pool.close(), not pool.terminate(). Terminate manually in that case:

from contextlib import closing

with closing(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
    pool.terminate()

or create your own terminating() context manager:

from contextlib import contextmanager

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

with terminating(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • 1
    Well Sir thank you for such a detailed explanation. You're solution has moved the function a step closer to working - now I am getting this error: `PicklingError: Can't pickle : attribute lookup __builtin__.function failed` – sidewaiise Sep 22 '14 at 10:38
  • @sidewaiise: are you trying to use a method on a class perhaps? See [Multiprocessing: using Pool.map on a function defined in a class](http://stackoverflow.com/q/3288595) – Martijn Pieters Sep 22 '14 at 10:40
  • Heh I was actually just reading that as you posted this. Thanks - I'll have a read and comment shortly. – sidewaiise Sep 22 '14 at 10:43
  • Not sure I understand the solution on that link - I'm not using classes. It's as simple as I wrote above, except that it is inside another function.... eg: **** Refer to Edited Question ^ **** So not sure.. any ideas? – sidewaiise Sep 22 '14 at 10:49
  • @sidewaiise: functions nested in another function are not picklable. Normally, pickle will store just a name (*`modulename.functionname`*) then import the same function on the other side. You cannot do that when you need to call another function to get the function. – Martijn Pieters Sep 22 '14 at 11:02
  • Getting there... new error: `IndexError: string index out of range`. Again no explanation on where the error actually occurs. – sidewaiise Sep 22 '14 at 12:24
  • (I've taken the Pool method out of the original function also) – sidewaiise Sep 22 '14 at 12:25
  • If you have new problems, better ask a new question. Your original issue here has been solved after all, and we now have a nice question + answer that'll perhaps others in future. No need to muddle this with new, different problems. – Martijn Pieters Sep 22 '14 at 12:33
  • True. Have done just now actually, thanks. http://stackoverflow.com/questions/25974256/python-multiprocessing-pool-map-throws-exception-indexerror-string-index-out – sidewaiise Sep 22 '14 at 12:34
  • Excuse me, but when I'm creating my own "terminating" do I have to do `from contextlib import contextmanager` ? – paulochf May 14 '15 at 20:57
  • 1
    @paulochf: for some reason the decorator line was missing. Only import it when you actually plan on using it! :-) (and *sorry*). – Martijn Pieters May 14 '15 at 21:01
1

with statement is for object that have __enter__ and __exit__ functions, i.e. Context Manager Types
multiprocessing.Pool is not Context Manager Type. try do the following:

pool = Pool(processes=2)
pool.map(myFunction, mylist)
pool.map(myfunction2, mylist2)
Elisha
  • 4,811
  • 4
  • 30
  • 46
  • 3
    The question is *why the OP thought it was*. The answer is that the Python 3 documentation tells you it is. But it also qualifies that as Python 3.3 and up. – Martijn Pieters Sep 22 '14 at 07:37