Python - Limiting the Number of Threads while passing arguments

Question

I am trying to run some threads using a thread limiter to keep number of threads to 10. I had an example to use as guide but I need to pass in some arguments to the function when the calling the thread. I am struggling with the passing of arguments. I marked with ### the areas where I am not sure the syntax and where I think my problem is.

I am trying to use this The right way to limit maximum number of threads running at once? as a guide. Here is the sample code I am trying to follow. My example is below that. Any time I try to pass in all the arguments I get back TypeError: __ init__() takes 1 to 6 arguments but 17 were passed In my example below I cut the arguments down to 4 to make it easier to read but I have 17 arguments in my live code. In example I keep the arguments down to run, main_path, target_path, jiranum to keep reading easier

threadLimiter = threading.BoundedSemaphore(maximumNumberOfThreads)

class MyThread(threading.Thread):

def run(self):  
    threadLimiter.acquire()
    try:
        self.Executemycode()
    finally:
        threadLimiter.release()

def Executemycode(self):
    print(" Hello World!") 
    # <your code here>

My code

import os
import sys
import threading

threadLimiter = threading.BoundedSemaphore(10)

class MyThread(threading.Thread):

  def run(self):  ### I also tried (run, main_path, target_path, jiranum)
     threadLimiter.acquire()
     try:
       self.run_compare(run, main_path, target_path, jiranum) #### I also tried self
     finally:
       threadLimiter.release()

  def run_compare(run, main_path, target_path, jiranum):   #### ???
    os.chdir(target_path)
    os.system(main_path + ', ' + target_path + ',' + jiranum + ',' + run)

if __name__ == '__main__':
    #set the needed variables
    threads = []

    for i in range (1, int(run)+1):
       process=threading.Thread(target=MyThread, args=(str(i), main_path, target_path, jiranum)) #### Is this defined right?
       process.start()
       threads.append(process)

       for process in threads:
          process.join()

score 1 · Accepted Answer · answered Sep 30 '21 at 21:53

This would probably be a simpler task with concurrent.futures but I like getting my hands dirty, so here we go. A few suggestions:

I find classes as thread targets often complicate things, so if there's no compelling reason, keep it simple
It's easier to use a with block to acquire and release a semaphore, and a regular semaphore usually suffices in that case
17 arguments can get messy; I would build a tuple of the arguments outside the call to threading.Thread() so it's easier to read, then unpack the tuple in the thread

This should work as a simple example; os.system() just echoes something and sleeps, so you can see the thread count is limited by the semaphore.

import os
import threading
from random import randint

threadLimiter = threading.Semaphore(10)

def run_config(*args):
    run, arg1, arg2 = args           # unpack the 17 args by name

    with threadLimiter:
        seconds = randint(2, 7)
        os.system(f"echo run {run}, args {arg1} {arg2} ; sleep {seconds}")

if __name__ == '__main__':
    threads = []
    run = "20"          # I guess this is a string because of below?

    for i in range (1, int(run)+1):
        thr_args = (str(i), "arg1",
                    "arg2")           # put the 17 args here
        thr = threading.Thread(target=run_config, args=thr_args)
        thr.start()
        threads.append(thr)

    for thr in threads:
        thr.join()

this worked very well.. Only issue i am having is when it first spools up it does through an error that it cant create a thread but it does run all the processes. thanks — personalt, Oct 02 '21 at 01:00

Python - Limiting the Number of Threads while passing arguments

1 Answers1