1

I am trying to do multiprocessing on some files. The files belong to two different categories namely A and B. The idea is to iterate over all the files where the filenames are contained in two lists category_A and category_B. Here is a simple class that I have written for doing multiprocessing:

class ImageProc:
    def __init__(self):
        self.data = []

    def __call__(self, sample, category="A"):
        subject = str(sample)
        sample = loadmat(sample)
        age  = int(sample["Age"][0][0])
        nb_images = sample['images'].shape[2]
        del sample
        for i in range(nb_images):
            self.data.append((subject, category, age, i))
        gc.collect()

# This works fine for category A
proc = ImageProc()
pool = Pool()
_ = pool.map(proc, category_A)

But now I want to use the same instance and call the same function for category_B for which I have to explicitly pass the argument category="B" in the __call__ method. Can anyone please help me how to achieve that?

EDIT: Given the useful comments, I also want to elaborate that this list, represented here by self.data, is common for both category_A and category_B. If I make it global, then it won't be possible use the pool instances on the same list to write data.

enterML
  • 2,110
  • 4
  • 26
  • 38
  • Depending on your use case, `pool.starmap(...)` may be an option for you, to achieve what you want. – JohanL Sep 30 '18 at 18:16
  • 1
    Possible duplicate of [Python multiprocessing pool.map for multiple arguments](https://stackoverflow.com/questions/5442910/python-multiprocessing-pool-map-for-multiple-arguments) – stovfl Sep 30 '18 at 18:17
  • @JohanL can you give an example? – enterML Sep 30 '18 at 18:21
  • @stovfl this one is similar but the implementation is totally different. Using same instance makes it a special case here I guess – enterML Sep 30 '18 at 18:22
  • You _cannot_ get the exact same instance when using `multiprocessing`. Each process will have its own instance, and that is irrespective of whether you use only category A or both A and B. – JohanL Sep 30 '18 at 18:28
  • @JohanL then how should I proceed? – enterML Sep 30 '18 at 18:29
  • That depends on what you want to achieve. I don't even know what you are trying to achieve or why you want to use the same instance in different processes. – JohanL Oct 01 '18 at 13:26

0 Answers0