Stack Overflow
Stack Overflow

Questions tagged [pathos]

'pathos' provides a fork of python's 'multiprocessing', where 'pathos.multiprocessing' can send a much broader range of the built-in python types across a parallel 'map' and 'pipe' (similar to python's 'map' and 'apply'). 'pathos' also provides a unified interface for parallelism across processors, threads, sockets (using a fork of 'parallelpython'), and across 'ssh'.

pathos provides a fork of python's multiprocessing, where pathos.multiprocessing can send a much broader range of the built-in python types across a parallel map and pipe (similar to python's map and apply). pathos also provides a unified interface for parallelism across processors, threads, sockets (using a fork of parallelpython), and across ssh.

190 questions
20
votes
2 answers

Python multiprocessing with pathos

I am trying to use Python's pathos to designate computations into separate processes in order to accelerate it with multicore processor. My code is organized like: class: def foo(self,name): ... setattr(self,name,something) ... def…
user3708829
  • 199
  • 1
  • 1
  • 4
17
votes
2 answers

How to read webcam in separate process on OSX?

I'm reading out a webcam on OSX, which works fine with this simple script: import cv2 camera = cv2.VideoCapture(0) while True: try: (grabbed, frame) = camera.read() # grab the current frame frame = cv2.resize(frame, (640, 480))…
kramer65
  • 50,427
  • 120
  • 308
  • 488
8
votes
1 answer

multiprocessing -> pathos.multiprocessing and windows

I'm currently using the standard multiprocessing in python to generate a bunch of processes that will run indefinitely. I'm not particularly concerned with performance; each thread is simply watching for a different change on the filesystem, and…
jdmcbr
  • 5,964
  • 6
  • 28
  • 38
7
votes
1 answer

TypeError: 'MapResult' object is not iterable using pathos.multiprocessing

I'm running a spell correction function on a dataset I have. I used from pathos.multiprocessing import ProcessingPool as Pool to do the job. Once the processing is done, I'd like to actually access the results. Here is my code: import codecs import…
Sam
  • 641
  • 1
  • 7
  • 17
7
votes
1 answer

Python multiprocessing, using pool multiple times in a loop gets stuck after first iteration

I have the following situation, where I create a pool in a for loop as follows (I know it's not very elegant, but I have to do this for pickling reasons). Assume that the pathos.multiprocessing is equivalent to python's multiprocessing library (as…
DaveTheAl
  • 1,995
  • 4
  • 35
  • 65
7
votes
2 answers

pathos: parallel processing options - Could someone explain the differences?

I am trying to run parallel processes under python (on ubuntu). I started using multiprocessing and it worked fine for simple examples. Then came the pickle error, and so I switched to pathos. I got a little confused with the different options and…
Olivier
  • 215
  • 2
  • 8
6
votes
2 answers

Multiprocessing in a loop, "Pool not running" error

I'm trying to run some calculation in loop, each calculation creates, uses and closes a pool. But the calculation only runs once and then throws an error: "Pool not running". Of course the old one is not running, but shouldn't the new one be…
Anna
  • 199
  • 1
  • 10
6
votes
1 answer

Pathos can't pickle SwigPyObject created by GDAL module

I have a class which opens a large raster image using the GDAL module (https://pypi.org/project/GDAL/) and extracts small images from it at a number of locations, defined by a passed list of coordinate tuples. I want to process a large list of…
liamvharris
  • 350
  • 3
  • 16
6
votes
0 answers

Can't pickle local object

I would like to use multiprocess package for example in this code. I've tried to call the function create_new_population and distribute the data to 8 processors, but when I do I get pickle error. Normally the function would run like this:…
Panos Kalatzantonakis
  • 12,525
  • 8
  • 64
  • 85
6
votes
0 answers

using pathos with ProcessPoolExecutor

its possible to use concurrent.futures (ProcessPoolExecutor) with pathos ? asking this because when i try to use: with concurrent.futures.ProcessPoolExecutor() as executor: ... executor.map() i will get: PicklingError: Can't pickle : attribute…
xampione
  • 71
  • 1
  • 6
6
votes
1 answer

multiprocessing numpy not defined error

I am using the following test code: from pathos.multiprocessing import ProcessingPool as Pool import numpy def foo(obj1, obj2): a = obj1**2 b = numpy.asarray(range(1,5)) return obj1, b if __name__ == '__main__': p = Pool(5) res =…
Zanam
  • 4,607
  • 13
  • 67
  • 143
5
votes
1 answer

Does pathos.multiprocessing have starmap?

I got an error when executing code below. The problem seems to be map doesn't support functions taking multiple inputs, just as in the python built-in multiprocessing package. But in the built-in package, there is a starmap that solves this issue.…
Indominus
  • 1,228
  • 15
  • 31
5
votes
0 answers

"could not load glyph" in multiprocessing

I'm running into an issue using pathos multiprocessing that runs figure saving function. from pathos.multiprocessing import ProcessingPool as Pool datasets = Pool().map(model_handler, analysisParams) which throws File…
Dae
  • 89
  • 1
  • 9
5
votes
0 answers

Pathos, Dask, futures, which one to use for parallel cluster application?

I am confused here. I have an application that is CPU bounded so I went to implementing a parallelisation using multiprocess to overcome GIL issues. I first tried to use multiprocessing and futures but I faced a pickling issue so I went to pathos…
tupui
  • 5,738
  • 3
  • 31
  • 52
4
votes
0 answers

Nested Multiprocessing Pool in Python using Pathos/Multiprocessing

I am trying to nest multiprocessing pool within another multiprocessing pool. Both levels need to be process pools, thread pools are not required for my purposes. However all 5 of my attempts using multiprocssing or pathos have failed Attempt…
Nyxynyx
  • 61,411
  • 155
  • 482
  • 830
1
2 3
…
12 13