IronPython Multiprocessing Module

Question

I have some dlls written in Dotnet which I am using to access thousands of binary files. The data is compartmentalized by directories, so as a performance enhancement I thought of using multiple processes or threads to churn through the files.

I have a function, currently part of the main class (requiring self as an argument), this could easily be refactored to a private method.

My first inclination is to use the Multiprocess module, but that doesn't seem to be available for IronPython.

My next thought was to use Task

def __createThreads(self):
    tasks = Array.CreateInstance(Task, 5)
    for idx in range(0, 5):
        tasks.append(Task.Factory.StartNew(self.__doWork, args=(idx,)))

    Task.WaitAll(tasks)

def __doWork(self, idx):

    for index in range (0, idx):
        print "Thread: %d | Index: %d" % (idx, index)

Or to use Thread

def __createThreads(self):
    threads = list()
    for idx in range(0, 5):
        t = Thread(ThreadStart(self.__doWork))
        t.Start()
        threads.append(t)

    while len(threads) > 0:
        time.sleep(.05)
        for t in threads:
            if(not t.IsAlive):
                threads.remove(t)

What I cannot find is a IronPython example of how to pass arguements

score 1 · Answer 1 · edited May 23 '17 at 12:20

1

Please be aware that your two examples are not exactly equivalent. The task version will only create/use actual concurrent threads when the run-time thinks it is a good idea (unless you specify TaskCreationOptions.LongRunning). You have to decide what works for your use-case.

The easiest way to pass the idx argument to the __doWork function would be using a lambda to capture the value and invocation. (please be aware of scoping issues as discussed in this question which also hints at alternative solutions for introducing an intermediate scope)

tasks.append(Task.Factory.StartNew(lambda idx = idx: self.__doWork(idx)))

As a side-note: You will have to convert your task list to an array in order for Task.WaitAll to be happy.

edited May 23 '17 at 12:20

Community

1
1

answered Sep 09 '13 at 12:07

Simon Opelt

6,136
2
35
66

Thanks, and that leads to the obvious next question which is how to specify the task as long running. The documentation doesn't include IronPython. – Bill Kidd Sep 09 '13 at 13:25
There is really nothing IronPython specific in terms of documentation. Except for a few corner cases you can just use all .net APIs as expected. [StartNew](http://msdn.microsoft.com/en-us/library/dd321263.aspx) just takes the creation options as the second argument which would look like `tasks.append(Task.Factory.StartNew(lambda idx = idx: self.__doWork(idx), TaskCreationOptions.LongRunning))` for sour sample. – Simon Opelt Sep 10 '13 at 18:26

IronPython Multiprocessing Module

1 Answers1