1

My current project requires using multiple processes. I need to share an array between those processes. The array needs to be able to be written to at any time. And the array has to have multiple dimensions. (example: [["test",2],[87209873,"howdy"]]) I've been looking for an answer to this for a few hours now, but I can't find anything. Please help. Thanks in advance!

WimpyLlama
  • 49
  • 1
  • 6
  • Yes, using the multiprocessing module. – WimpyLlama Mar 17 '20 at 19:16
  • You could look into [shared memory](https://docs.python.org/3/library/multiprocessing.shared_memory.html). – Fred Larson Mar 17 '20 at 19:18
  • Does this answer your question? [Shared memory in multiprocessing](https://stackoverflow.com/questions/14124588/shared-memory-in-multiprocessing) – Fred Larson Mar 17 '20 at 19:22
  • Not quite. I just need a simple explanation. I've looked into using "Array()", but I'm not sure how to use it with 2D arrays. – WimpyLlama Mar 17 '20 at 19:29
  • 1) I think in shared memory, you probably can't grow an array; it's a pretty different thing from other Python types. 2) To get 2 dimensions in your array, do something inspired by what a C compiler does: linear_index = two_d.row * 10 + two_d.column. 3) if you can use a https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Array that's probably a good bet. – dstromberg Mar 17 '20 at 20:49

2 Answers2

1

Try it:

from multiprocessing import Pool, Manager                                         

def worker(v, array):                                                             
    array.append(["test", v])                                                     

def main():                                                                       
    foo = [["test", 2], [87209873, "howdy"]]                                      

    array = Manager().list(foo)                                                   

    with Pool(processes=4) as pool:                                               
        pool.starmap(worker, [(i, array) 
            for i in range(4)])                      

    print(array)                                                                  

if __name__ == "__main__":                                                        
    main()

[EDITED]

If you want, that the main program keeps running, during calculating, wrap pooling in a separate thread: 

from multiprocessing import Pool, Manager                                         
from threading import Thread                                                      

def _worker(v, array):                                                            
    for i in range(10000):                                                        
        array.append(["test", v])                                                 

def processor(array):                                                             
    with Pool(processes=4) as pool:                                               
        pool.starmap(_worker, [(i, array) 
            for i in range(4)])                     

def main():                                                                       
    foo = [["test", 2], [87209873, "howdy"]]                                      
    array = Manager().list(foo)                                                   
    t = Thread(target=processor, args=(array,))                                   

    t.start()                                                                     

    print("Good day!")    

    # Wait, while thread ends.
    # Without doing it, you'll print array, 
    # not knowing when the thread ended.
    t.join()

    print(array)                                                    

if __name__ == "__main__":                                                        
    main()
Kassi
  • 151
  • 2
  • 12
  • The problem with this is that it pauses the program. I need the main program to keep running as a few others run in the background. – WimpyLlama Mar 17 '20 at 20:49
  • Now I'm getting "[WinError 10053] An established connection was aborted by the software in your host machine" Not sure why or what this even means. – WimpyLlama Mar 17 '20 at 21:30
  • I've got the error to go away, but the list is still not spreading across the processes. – WimpyLlama Mar 17 '20 at 22:19
  • @WimpyLlama, yes, exactly - I forgot use `t.join()`, that wait for thread end. Can't check code on VS. Hm... It's strange, that you get error. Don't think, that it associated with code... – Kassi Mar 17 '20 at 23:17
0

First of all, a list is not an array, if you want to share a list between different processes you can use a Manager from the multiprocessing module, for example:

import multiprocessing as mp


def remove_last_element(mp_list: list):
    mp_list.pop()


def append_list(mp_list: list):
    mp_list.append([12, 'New Hello'])


if __name__ == "__main__":
    mp_list = mp.Manager().list()
    mp_list.append(['Hello'])
    print("before multiprocessing:", mp_list)

    worker1 = mp.Process(target=remove_last_element, args=(mp_list,))
    worker2 = mp.Process(target=append_list, args=(mp_list,))

    worker1.start()
    worker2.start()

    worker1.join()
    worker2.join()

    print("after multiprocessing:", mp_list)

>>> before multiprocessing: [['Hello']]
>>> after multiprocessing: [[12, 'New Hello']]
marcos
  • 4,473
  • 1
  • 10
  • 24
  • I've used this method before, but for some odd reason, it does not let me append to a list. For example, I can't do "testList[1].append("hello")" It just does nothing. If I don't use manager, the appending works just fine. – WimpyLlama Mar 17 '20 at 19:57
  • @WimpyLlama what exactly can't u do? – marcos Mar 17 '20 at 20:01
  • from multiprocessing import * test = Manager().list() test.append(["test","data"]) test[0].append("is cool") #This code just doesn't work – WimpyLlama Mar 17 '20 at 20:05
  • Try it out yourself and you'll see – WimpyLlama Mar 17 '20 at 20:05