0

I am calling full_item() for each element of a list size_DF and passing some parameters to the function.

map(lambda x : object.full_item(sc, Database, len(x), end_date),size_DF)

map function is working but it is taking more time as list contains more elements.

I tried the multiprocessing but getting the error as below:

from multiprocessing import Pool

pool = Pool()

pool.map(lambda x : object.full_item(sc, Database, len(x), end_date),size_DF)

cPickle.PicklingError: Can't pickle : attribute lookup builtin.function failed

Please help and let me know how can i proceed.

desertnaut
  • 57,590
  • 26
  • 140
  • 166
A.N.Gupta
  • 331
  • 1
  • 5
  • 16
  • 1
    Possible duplicate of [Compulsory usage of if \_\_name\_\_=="\_\_main\_\_" in windows while using multiprocessing](https://stackoverflow.com/questions/20360686/compulsory-usage-of-if-name-main-in-windows-while-using-multiprocessi) – Ma0 Feb 28 '18 at 14:28
  • all multiprocessing stuff have to be inside the `if __name__=='__main__':` block – Ma0 Feb 28 '18 at 14:28
  • 1
    @Ev.Kounis, yes it's true for windows machines, but it's not the issue here. The issue is that multiprocessing module cannot serialize one of OP objects. – Jean-François Fabre Feb 28 '18 at 14:40
  • I have put all the stuff inside if __name__=='__main__': block but still getting the same issue. – A.N.Gupta Feb 28 '18 at 15:00

0 Answers0